Nov 02 13:56:34.744097 ip-10-0-134-9 systemd[1]: Starting Kubernetes Kubelet... Nov 02 13:56:35.102666 ip-10-0-134-9 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:56:35.102666 ip-10-0-134-9 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 13:56:35.102666 ip-10-0-134-9 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:56:35.102666 ip-10-0-134-9 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:56:35.105925 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105858 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.105925 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105925 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105931 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105935 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105937 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105940 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105943 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105946 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105949 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105952 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105954 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105957 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105959 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105962 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105964 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105967 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105975 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105978 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105980 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105983 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105986 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.106002 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105988 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105991 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105993 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.105996 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106000 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106004 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106007 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106010 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106013 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106016 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106018 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106021 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106024 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106027 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106029 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106032 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106034 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106036 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.106460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106039 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106041 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106044 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106046 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106049 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106051 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106054 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106056 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106058 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106061 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106064 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106067 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106069 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106072 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106074 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106078 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106081 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106084 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106086 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106089 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.106919 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106092 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106094 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106097 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106099 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106102 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106104 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106107 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106109 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106112 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106114 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106117 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106119 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106122 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106124 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106127 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106130 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106132 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106135 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106137 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.107386 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106140 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106142 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106144 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106147 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106150 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106153 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106155 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106158 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106160 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106163 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106166 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106169 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106171 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106174 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106176 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106179 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106181 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106184 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106186 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106190 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.107838 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106192 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106195 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106197 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106200 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106202 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106205 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106207 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.106209 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107845 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107853 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107856 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107859 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107861 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107864 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107867 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107869 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107872 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107878 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107881 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107883 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.108322 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107886 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107889 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107892 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107894 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107897 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107899 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107902 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107904 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107907 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107909 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107912 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107916 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107919 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107922 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107926 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107929 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107932 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107935 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107937 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.108854 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107940 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107943 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107945 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107948 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107950 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107953 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107956 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107958 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107961 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107963 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107966 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107968 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107971 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107974 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107976 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107978 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107981 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107984 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107986 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.109304 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107989 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107991 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107995 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.107997 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108000 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108002 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108005 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108007 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108010 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108013 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108016 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108018 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108021 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108023 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108026 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108028 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108031 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108033 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108035 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108038 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.109766 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108040 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108043 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108045 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108048 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108051 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108054 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108056 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108059 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108061 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108064 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108068 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108072 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108075 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108079 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108081 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108085 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108087 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108090 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108093 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.110258 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108095 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108098 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108101 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108103 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108106 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108108 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108111 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108113 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108116 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108118 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108121 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108124 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108126 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108130 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108132 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108134 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108137 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108612 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108617 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.110794 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108620 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108623 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108625 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108628 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108631 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108635 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108639 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108642 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108645 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108647 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108650 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108652 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108655 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108658 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108660 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108663 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108665 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108668 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108670 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.111444 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108673 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108676 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108678 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108681 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108684 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108687 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108689 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108692 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108695 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108697 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108700 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108702 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108705 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108708 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108711 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108713 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108716 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108718 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108721 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108723 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.111966 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108726 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108729 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108731 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108733 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108736 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108738 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108741 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108743 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108746 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108748 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108751 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108753 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108756 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108759 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108761 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108767 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108770 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108788 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108791 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.112464 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108793 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108796 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108798 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108801 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108803 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108807 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108809 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108816 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108819 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108822 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108824 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108827 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108829 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108832 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108834 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108837 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108839 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108842 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108845 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.112968 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108849 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108853 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108856 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108858 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108861 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108864 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108866 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108869 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108871 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108876 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108879 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108881 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108884 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108886 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108889 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108892 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108894 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108897 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108899 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.113415 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108902 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108905 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108908 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108910 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108914 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108916 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108919 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.108921 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.109997 2583 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110007 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110013 2583 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110017 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110022 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110025 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110031 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110036 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110040 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110043 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110047 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110050 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110053 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110056 2583 flags.go:64] FLAG: --cgroup-root="" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110061 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 13:56:35.113900 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110064 2583 flags.go:64] FLAG: --client-ca-file="" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110067 2583 flags.go:64] FLAG: --cloud-provider="external" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110070 2583 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110074 2583 flags.go:64] FLAG: --cluster-domain="" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110077 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110080 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110083 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110087 2583 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110091 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110094 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110097 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110100 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110103 2583 flags.go:64] FLAG: --contention-profiling="false" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110106 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110109 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110113 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110115 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110120 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110123 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110125 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110128 2583 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110131 2583 flags.go:64] FLAG: --enable-server="true" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110134 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110139 2583 flags.go:64] FLAG: --event-burst="100" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110142 2583 flags.go:64] FLAG: --event-qps="50" Nov 02 13:56:35.114428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110145 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110148 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110151 2583 flags.go:64] FLAG: --eviction-hard="" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110154 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110157 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110160 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110165 2583 flags.go:64] FLAG: --eviction-soft="" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110168 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110171 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110174 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110177 2583 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110180 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110182 2583 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110185 2583 flags.go:64] FLAG: --feature-gates="" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110189 2583 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110192 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110195 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110198 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110201 2583 flags.go:64] FLAG: --healthz-port="10248" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110204 2583 flags.go:64] FLAG: --help="false" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110207 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110210 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110214 2583 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 13:56:35.115127 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110216 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110219 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110223 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110226 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110228 2583 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110231 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110234 2583 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110237 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110240 2583 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110243 2583 flags.go:64] FLAG: --kube-reserved="" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110246 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110249 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110252 2583 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110254 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110257 2583 flags.go:64] FLAG: --lock-file="" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110262 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110265 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110268 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110273 2583 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110276 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110279 2583 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110282 2583 flags.go:64] FLAG: --logging-format="text" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110285 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110289 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 13:56:35.115666 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110292 2583 flags.go:64] FLAG: --manifest-url="" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110295 2583 flags.go:64] FLAG: --manifest-url-header="" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110299 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110302 2583 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110306 2583 flags.go:64] FLAG: --max-pods="110" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110309 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110312 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110315 2583 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110318 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110321 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110324 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110327 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110333 2583 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110336 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110339 2583 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110342 2583 flags.go:64] FLAG: --pod-cidr="" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110345 2583 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110348 2583 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110351 2583 flags.go:64] FLAG: --pods-per-core="0" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110354 2583 flags.go:64] FLAG: --port="10250" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110357 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110360 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-063e0fc4fb4fe3492" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110363 2583 flags.go:64] FLAG: --qos-reserved="" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110366 2583 flags.go:64] FLAG: --read-only-port="10255" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110369 2583 flags.go:64] FLAG: --register-node="true" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110372 2583 flags.go:64] FLAG: --register-with-taints="" Nov 02 13:56:35.116248 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110376 2583 flags.go:64] FLAG: --registry-burst="10" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110379 2583 flags.go:64] FLAG: --registry-qps="5" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110381 2583 flags.go:64] FLAG: --reserved-cpus="" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110384 2583 flags.go:64] FLAG: --reserved-memory="" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110388 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110391 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110394 2583 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110397 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110400 2583 flags.go:64] FLAG: --runonce="false" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110402 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110405 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110409 2583 flags.go:64] FLAG: --seccomp-default="false" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110411 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110415 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110418 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110421 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110424 2583 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110426 2583 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110429 2583 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110432 2583 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110435 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110438 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110441 2583 flags.go:64] FLAG: --system-cgroups="" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110445 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110450 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 13:56:35.116906 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110453 2583 flags.go:64] FLAG: --tls-cert-file="" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110456 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110460 2583 flags.go:64] FLAG: --tls-min-version="" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110463 2583 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110465 2583 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110468 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110471 2583 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110474 2583 flags.go:64] FLAG: --v="2" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110478 2583 flags.go:64] FLAG: --version="false" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110482 2583 flags.go:64] FLAG: --vmodule="" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110486 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.110490 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112022 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112039 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112043 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112050 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112054 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112060 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112081 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112091 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112097 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112103 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.117572 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112122 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112279 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112284 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112288 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112291 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112293 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112296 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112299 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112302 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112304 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112308 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112311 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112313 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112316 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112319 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112321 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112324 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112327 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112330 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.118121 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112333 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112336 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112339 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112342 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112345 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112347 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112350 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112353 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112355 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112362 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112365 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112367 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112370 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112373 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112376 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112379 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112382 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112384 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112389 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.118565 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112394 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112397 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112400 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112402 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112405 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112408 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112411 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112414 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112416 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112419 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112421 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112424 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112427 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112429 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112432 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112434 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112437 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112440 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112442 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112445 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.119047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112448 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112450 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112453 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112455 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112458 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112460 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112463 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112466 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112469 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112471 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112474 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112477 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112480 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112482 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112486 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112491 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112494 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112497 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112500 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.119523 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112502 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112505 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112508 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112511 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112513 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112516 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112518 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112521 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112524 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112526 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112529 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112532 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112534 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112537 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112540 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112542 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112545 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112547 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.120033 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112550 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.112584 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112624 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112629 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112633 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112635 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112638 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112641 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112645 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112648 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112651 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112654 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112657 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112659 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112662 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112665 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.120460 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112667 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112670 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112672 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112675 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112677 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112680 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112682 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112685 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112688 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112691 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112693 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112696 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112698 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112701 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112703 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112706 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112708 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112711 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112716 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112719 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.120908 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112722 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112724 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112727 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112729 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112732 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112735 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112737 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112740 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112743 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112746 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112748 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112751 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112754 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112756 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112759 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112761 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112764 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112766 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112768 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.121394 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112771 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112786 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112789 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112791 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112794 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112796 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112800 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112803 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112806 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112808 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112811 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112815 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112819 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112822 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112824 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112827 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112829 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112832 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112835 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112837 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.121864 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112840 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112843 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112846 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112849 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112852 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112854 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112857 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112859 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112862 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112864 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112867 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112869 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112872 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112874 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112877 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112879 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112882 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112884 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112887 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.122324 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112889 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112892 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112895 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112900 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112904 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112907 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112910 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112912 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112915 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112918 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112920 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112923 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112925 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.112928 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.119907 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.119919 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.120657 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.120663 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120716 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.122770 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120721 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120725 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120727 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120730 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120733 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120736 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120739 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120741 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120744 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120747 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120750 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120752 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120755 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120758 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120760 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120763 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120765 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120768 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120770 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120788 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.123233 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120793 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120795 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120798 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120801 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120804 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120807 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120810 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120812 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120817 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120820 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120823 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120825 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120828 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120831 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120834 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120837 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120839 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120842 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120844 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.123696 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120847 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120849 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120852 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120855 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120857 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120859 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120862 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120865 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120867 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120870 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120872 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120875 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120877 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120880 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120882 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120886 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120891 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120895 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120898 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120901 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.124157 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120903 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120907 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120911 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120914 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120916 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120920 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120924 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120927 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120929 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120932 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120935 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120937 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120940 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120942 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120945 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120947 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120950 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120952 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120955 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.124627 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120958 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120960 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120963 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120965 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120968 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120970 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120973 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120976 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120978 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120980 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120983 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120986 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120988 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120991 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120994 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120996 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.120999 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121002 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121005 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121007 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.125163 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121010 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121012 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121015 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121017 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121020 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121023 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121025 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.121052 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121179 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121185 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121188 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121191 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121194 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121197 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121200 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121202 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.125660 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121205 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121208 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121211 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121215 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121218 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121221 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121224 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121226 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121229 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121232 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121234 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121237 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121240 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121243 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121246 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121249 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121251 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121254 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121256 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121259 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.126047 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121262 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121264 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121267 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121269 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121273 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121275 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121278 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121280 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121283 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121285 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121288 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121290 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121293 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121295 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121298 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121300 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121302 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121305 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121308 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.126512 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121310 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121313 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121315 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121318 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121321 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121323 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121326 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121329 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121331 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121334 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121336 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121339 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121341 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121344 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121346 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121349 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121351 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121354 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121357 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.126971 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121359 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121362 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121364 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121367 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121369 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121372 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121374 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121377 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121379 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121382 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121384 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121387 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121389 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121392 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121394 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121397 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121400 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121402 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121405 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121408 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.127434 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121410 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121413 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121416 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121418 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121421 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121423 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121426 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121428 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121431 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121433 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121437 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121441 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121444 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121447 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121450 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121453 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121455 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121458 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121460 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.127903 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121463 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.121485 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121515 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121520 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121524 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121528 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121531 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121534 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121537 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121540 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121543 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121545 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121548 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121552 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121555 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121558 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:56:35.128347 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121561 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121564 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121566 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121568 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121571 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121574 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121577 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121581 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121584 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121586 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121589 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121591 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121594 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121596 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121599 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121603 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121606 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121608 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121611 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:56:35.128717 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121613 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121616 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121618 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121621 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121623 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121626 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121629 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121631 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121634 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121636 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121640 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121643 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121645 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121648 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121651 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121653 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121656 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121658 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121661 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:56:35.129222 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121663 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121666 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121668 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121671 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121673 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121676 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121678 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121680 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121683 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121685 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121688 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121690 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121693 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121695 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121698 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121701 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121703 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121706 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121708 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:56:35.129662 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121711 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121713 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121716 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121718 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121722 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121725 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121727 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121730 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121733 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121735 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121738 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121740 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121743 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121745 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121748 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121750 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121753 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121755 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121758 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:56:35.130113 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121761 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121763 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121766 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121768 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121771 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121787 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121790 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121793 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121795 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121798 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121801 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121803 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121807 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121809 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121812 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:35.121814 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.121865 2583 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.124248 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.125303 2583 server.go:1014] "Starting client certificate rotation" Nov 02 13:56:35.130561 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.125396 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:56:35.131017 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.125429 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:56:35.131017 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.127587 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:56:35.132720 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.132703 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:56:35.147524 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.147496 2583 log.go:25] "Validated CRI v1 runtime API" Nov 02 13:56:35.151167 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.151152 2583 log.go:25] "Validated CRI v1 image API" Nov 02 13:56:35.152638 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.152609 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 13:56:35.156115 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.156099 2583 fs.go:135] Filesystem UUIDs: map[205ba9d1-56f0-45df-8ec7-55033244e160:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fea383b2-9528-482d-adfc-98b01e13307e:/dev/nvme0n1p4] Nov 02 13:56:35.156163 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.156116 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 13:56:35.160437 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.160334 2583 manager.go:217] Machine: {Timestamp:2026-11-02 13:56:35.159887901 +0000 UTC m=+0.315393814 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096811 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25677aeede08c6102c79586874c11a SystemUUID:ec25677a-eede-08c6-102c-79586874c11a BootID:e9b10335-2494-48cc-9064-18e37c02197c Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:4d:c6:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:4d:c6:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:2d:19:89:4c:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 13:56:35.161263 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.161252 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 13:56:35.161333 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.161322 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 13:56:35.164917 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.164890 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 13:56:35.165034 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.164919 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 13:56:35.165081 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.165037 2583 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 13:56:35.165081 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.165045 2583 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 13:56:35.165134 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.165082 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:56:35.165134 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.165100 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:56:35.166583 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.166573 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 13:56:35.171265 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.171253 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 13:56:35.171369 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.171360 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 13:56:35.172169 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.172160 2583 kubelet.go:493] "Attempting to sync node with API server" Nov 02 13:56:35.172199 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.172173 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 13:56:35.172199 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.172183 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 13:56:35.172199 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.172191 2583 kubelet.go:405] "Adding apiserver pod source" Nov 02 13:56:35.172199 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.172199 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 13:56:35.172493 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.172482 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:56:35.175056 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.175039 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 13:56:35.182339 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.182323 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:56:35.183240 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.183226 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 13:56:35.183303 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.183265 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 13:56:35.183446 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.183433 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 13:56:35.183510 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.183453 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 13:56:35.183510 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.183462 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 13:56:35.183510 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.183472 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 13:56:35.183510 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.183483 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 13:56:35.183510 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.183491 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 13:56:35.184119 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.184109 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 13:56:35.184167 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.184122 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 13:56:35.184167 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.184137 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 13:56:35.184167 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.184148 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 13:56:35.184167 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.184167 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 13:56:35.184347 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.184182 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 13:56:35.184936 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.184926 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 13:56:35.184993 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.184940 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 13:56:35.188555 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.188541 2583 server.go:1263] "Started kubelet" Nov 02 13:56:35.188905 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.188873 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 13:56:35.189220 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.188655 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 13:56:35.189325 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.189238 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 13:56:35.189233 ip-10-0-134-9 systemd[1]: Started Kubernetes Kubelet. Nov 02 13:56:35.191842 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.191828 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 13:56:35.192495 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.192484 2583 server.go:317] "Adding debug handlers to kubelet server" Nov 02 13:56:35.198094 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.198056 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 13:56:35.198428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.198405 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 13:56:35.199093 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.199079 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 13:56:35.199317 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.199293 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:35.199952 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.199932 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 13:56:35.199952 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.199934 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 13:56:35.200092 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.199964 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 13:56:35.200092 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.199993 2583 factory.go:55] Registering systemd factory Nov 02 13:56:35.200092 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.200008 2583 factory.go:223] Registration of the systemd container factory successfully Nov 02 13:56:35.200092 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.200027 2583 reconstruct.go:97] "Volume reconstruction finished" Nov 02 13:56:35.200092 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.200036 2583 reconciler.go:29] "Reconciler: start to sync state" Nov 02 13:56:35.200278 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.200207 2583 factory.go:153] Registering CRI-O factory Nov 02 13:56:35.200278 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.200220 2583 factory.go:223] Registration of the crio container factory successfully Nov 02 13:56:35.200278 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.200263 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 13:56:35.200430 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.200281 2583 factory.go:103] Registering Raw factory Nov 02 13:56:35.200430 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.200294 2583 manager.go:1199] Started watching for new ooms in manager Nov 02 13:56:35.201415 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.201402 2583 manager.go:319] Starting recovery of all containers Nov 02 13:56:35.212824 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.211257 2583 manager.go:324] Recovery completed Nov 02 13:56:35.216663 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.216645 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.218613 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.218593 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.218691 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.218669 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.218691 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.218682 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.219225 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.219206 2583 cpu_manager.go:226] "Starting" policy="none" Nov 02 13:56:35.219225 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.219223 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 13:56:35.219338 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.219227 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.219338 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.219240 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 13:56:35.220013 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.219994 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.220096 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.220027 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.220096 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.220038 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.221371 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.221339 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 13:56:35.221371 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.221341 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 13:56:35.221497 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.221398 2583 policy_none.go:50] "Start" Nov 02 13:56:35.221497 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.221410 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 13:56:35.221497 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.221420 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 13:56:35.223333 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.223319 2583 policy_none.go:44] "Start" Nov 02 13:56:35.225932 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.225910 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 13:56:35.226105 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.226093 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b6mh5" Nov 02 13:56:35.233281 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.232187 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-9.ec2.internal.18e43efa077a05a2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-9.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-9.ec2.internal,},FirstTimestamp:2026-11-02 13:56:35.18848349 +0000 UTC m=+0.343989402,LastTimestamp:2026-11-02 13:56:35.18848349 +0000 UTC m=+0.343989402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-9.ec2.internal,}" Nov 02 13:56:35.239213 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.239185 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 13:56:35.239213 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.239188 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 13:56:35.239342 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.239189 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 13:56:35.239588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.239571 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b6mh5" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.258366 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.262381 2583 manager.go:341] "Starting Device Plugin manager" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.262406 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.262415 2583 server.go:87] "Starting device plugin registration server" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.262595 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.262603 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.262736 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.262826 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.262834 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.264332 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.264379 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.264810 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.265901 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.265918 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.266540 2583 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 13:56:35.274428 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.266578 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 13:56:35.282500 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.282480 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:35.363653 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.363595 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.364410 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.364394 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.364494 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.364422 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.364494 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.364432 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.364494 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.364453 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.367542 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.367523 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal"] Nov 02 13:56:35.367641 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.367574 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.368290 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.368274 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.368353 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.368301 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.368353 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.368311 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.370278 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.370267 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.370417 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.370403 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.370460 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.370431 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.370960 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.370944 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.371032 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.370945 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.371032 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.370975 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.371032 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.370989 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.371032 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.370995 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.371032 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.371007 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.372928 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.372915 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.372983 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.372946 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:35.373577 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.373565 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:35.373630 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.373588 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:35.373630 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.373598 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:35.386147 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.386025 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.386235 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.386154 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-9.ec2.internal\": node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:35.401468 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.401448 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4f9ec4cb1fc6480c545594116d855c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"6e4f9ec4cb1fc6480c545594116d855c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.401539 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.401472 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.401539 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.401490 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.413309 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.413292 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-9.ec2.internal\" not found" node="ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.419700 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.419681 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-9.ec2.internal\" not found" node="ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.421614 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.421599 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:35.501702 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.501681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.501812 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.501705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.501812 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.501721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4f9ec4cb1fc6480c545594116d855c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"6e4f9ec4cb1fc6480c545594116d855c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.501812 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.501796 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4f9ec4cb1fc6480c545594116d855c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"6e4f9ec4cb1fc6480c545594116d855c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.501812 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.501802 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.501942 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.501807 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.521818 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.521798 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:35.622250 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.622202 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:35.718448 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.718428 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.722369 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.722354 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:35.724637 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:35.724628 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:35.823036 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.823014 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:35.923469 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:35.923449 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:36.024068 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:36.024047 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:36.124326 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:36.124306 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:36.125407 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.125391 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 13:56:36.224814 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:36.224763 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:36.243189 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:36.243164 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4f9ec4cb1fc6480c545594116d855c.slice/crio-7d17e915d5bb8a4853e0c98aa85a36c9b32e677facd1745e8e8417a902586bd3 WatchSource:0}: Error finding container 7d17e915d5bb8a4853e0c98aa85a36c9b32e677facd1745e8e8417a902586bd3: Status 404 returned error can't find the container with id 7d17e915d5bb8a4853e0c98aa85a36c9b32e677facd1745e8e8417a902586bd3 Nov 02 13:56:36.245419 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.245399 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:51:35 +0000 UTC" deadline="2028-06-19 00:28:09.632755745 +0000 UTC" Nov 02 13:56:36.245467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.245419 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14266h31m33.387339152s" Nov 02 13:56:36.253571 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.253554 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 13:56:36.265905 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:36.265881 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b7d7036f47f423ed3f88aa11798947.slice/crio-870177ed3be98157021524efd04da0aa1fd4bdedfd6e5f652f23634f2774d46d WatchSource:0}: Error finding container 870177ed3be98157021524efd04da0aa1fd4bdedfd6e5f652f23634f2774d46d: Status 404 returned error can't find the container with id 870177ed3be98157021524efd04da0aa1fd4bdedfd6e5f652f23634f2774d46d Nov 02 13:56:36.269288 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.269251 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"04b7d7036f47f423ed3f88aa11798947","Type":"ContainerStarted","Data":"870177ed3be98157021524efd04da0aa1fd4bdedfd6e5f652f23634f2774d46d"} Nov 02 13:56:36.270197 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.270172 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" event={"ID":"6e4f9ec4cb1fc6480c545594116d855c","Type":"ContainerStarted","Data":"7d17e915d5bb8a4853e0c98aa85a36c9b32e677facd1745e8e8417a902586bd3"} Nov 02 13:56:36.325829 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:36.325802 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:36.421562 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.421536 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:56:36.423446 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.423429 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:56:36.423542 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.423461 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:56:36.423542 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.423477 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:56:36.426807 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:36.426774 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:36.527426 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:36.527373 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:36.585730 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.585706 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:36.628157 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:36.628139 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:56:36.680174 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.680143 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:36.699506 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.699485 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:56:36.731442 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.731419 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:56:36.732667 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.732646 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:56:36.762691 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.762672 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:56:36.797097 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:36.797041 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:56:37.172968 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.172941 2583 apiserver.go:52] "Watching apiserver" Nov 02 13:56:37.200368 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.200332 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 13:56:37.212760 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.212736 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 13:56:37.213123 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.213099 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-5q6hs","kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal","openshift-dns/node-resolver-kjcwb","openshift-multus/multus-w8qtw","openshift-image-registry/node-ca-dtc82","openshift-multus/multus-additional-cni-plugins-wpzw4","openshift-multus/network-metrics-daemon-bjbp9","openshift-network-diagnostics/network-check-target-tqb8d","openshift-ovn-kubernetes/ovnkube-node-zx5gw","kube-system/global-pull-secret-syncer-ndjmx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72","openshift-cluster-node-tuning-operator/tuned-pv4lc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal","openshift-network-operator/iptables-alerter-r7fcg"] Nov 02 13:56:37.216100 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.216078 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:37.216203 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.216159 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:37.218571 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.218546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.221068 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.221038 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.227189 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.227168 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.229581 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.229553 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.232079 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.232057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.235800 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.234576 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.237195 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.237164 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.238412 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.238370 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.238552 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.238528 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.238632 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.238577 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 13:56:37.238706 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.238666 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mgxc\"" Nov 02 13:56:37.238762 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.238745 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.239674 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.239654 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:37.239803 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.239714 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:37.243499 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.242199 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.243499 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.242260 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:37.244264 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.244244 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.244428 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.244408 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m5vzv\"" Nov 02 13:56:37.244537 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.244495 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.244838 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.244818 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2zrsm\"" Nov 02 13:56:37.244941 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.244836 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 13:56:37.244941 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.244850 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 13:56:37.245046 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.244963 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.245171 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.245149 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:37.246990 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.246973 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.249024 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.247970 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.249024 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.248592 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlrs4\"" Nov 02 13:56:37.263422 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.263005 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 13:56:37.263422 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.263083 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-82d2t" Nov 02 13:56:37.301140 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.301120 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 13:56:37.313008 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.312981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-multus\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.313136 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313017 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-systemd-units\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.313136 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.313136 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-tmp-dir\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.313302 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313145 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-multus-certs\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.313302 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313203 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.313302 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-kubelet-config\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.313302 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-var-lib-kubelet\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.313506 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313312 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.313506 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-hosts-file\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.313506 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.313506 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313390 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-conf-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.313506 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313438 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-etc-kubernetes\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.313506 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-systemd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.313506 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313490 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysconfig\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313511 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-systemd\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313528 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-hostroot\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313545 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxw25\" (UniqueName: \"kubernetes.io/projected/fdee392a-98f2-4018-ba57-23a7bc2952b7-kube-api-access-rxw25\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3206ad3-d2a7-4eec-81cb-db3a09dee305-serviceca\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313594 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313651 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-lib-modules\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313675 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gwq\" (UniqueName: \"kubernetes.io/projected/f9a86134-c854-474c-8a22-8d9732ee781a-kube-api-access-c5gwq\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-config\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-script-lib\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313737 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrs9\" (UniqueName: \"kubernetes.io/projected/a3206ad3-d2a7-4eec-81cb-db3a09dee305-kube-api-access-fsrs9\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313762 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-modprobe-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313784 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-etc-selinux\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313808 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-k8s-cni-cncf-io\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.313877 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-log-socket\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfc9n\" (UniqueName: \"kubernetes.io/projected/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-kube-api-access-hfc9n\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313921 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-dbus\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313944 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-cni-binary-copy\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313972 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.313994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3206ad3-d2a7-4eec-81cb-db3a09dee305-host\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-binary-copy\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314037 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-konnectivity-ca\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314141 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-node-log\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314161 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovn-node-metrics-cert\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d3607406-fa52-4ff4-a4e3-9a41a050d66e-iptables-alerter-script\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314199 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314217 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-socket-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-kubelet\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314270 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-env-overrides\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314298 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.314673 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314321 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-system-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314338 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-netns\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-kubelet\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314388 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3607406-fa52-4ff4-a4e3-9a41a050d66e-host-slash\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314581 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-tuned\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-sys-fs\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314712 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g724w\" (UniqueName: \"kubernetes.io/projected/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-kube-api-access-g724w\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314748 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-cnibin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314790 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-netd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314818 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-kubernetes\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314877 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-tmp\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314906 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-slash\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314931 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.314960 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-agent-certs\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315017 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-conf\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315043 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-sys\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315070 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbp5h\" (UniqueName: \"kubernetes.io/projected/2063b915-4bd7-424a-a9d1-8d8065c79166-kube-api-access-lbp5h\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.315472 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-netns\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315120 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-ovn\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315141 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-bin\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cnibin\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315192 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-host\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315307 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-socket-dir-parent\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znb5\" (UniqueName: \"kubernetes.io/projected/d3607406-fa52-4ff4-a4e3-9a41a050d66e-kube-api-access-6znb5\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315352 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-run\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315369 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xk87\" (UniqueName: \"kubernetes.io/projected/75eff6b2-9975-4809-a676-f78efa1d45fa-kube-api-access-6xk87\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-registration-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315405 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-device-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-daemon-config\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315461 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-system-cni-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315489 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscx7\" (UniqueName: \"kubernetes.io/projected/60f37ff5-b4ee-46b6-a28c-7db9091eef76-kube-api-access-bscx7\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315518 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-bin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.316336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-var-lib-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.317147 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-etc-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.317147 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.315600 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:37.347690 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.347468 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.347690 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.347530 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.347690 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.347564 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.347690 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.347586 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 13:56:37.347690 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.347649 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 13:56:37.347690 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.347679 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:56:37.348090 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.347826 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cd6m4\"" Nov 02 13:56:37.348090 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.347939 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gjwh\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348260 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348278 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348284 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348298 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348327 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348342 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4wpm\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348346 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-75928\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348358 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348387 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348397 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348413 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 13:56:37.348467 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348442 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 13:56:37.349128 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348454 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kftg9\"" Nov 02 13:56:37.349128 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.348529 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 13:56:37.416169 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416138 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-system-cni-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.416299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscx7\" (UniqueName: \"kubernetes.io/projected/60f37ff5-b4ee-46b6-a28c-7db9091eef76-kube-api-access-bscx7\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.416299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-bin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.416299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416216 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-system-cni-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.416299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-var-lib-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.416299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-etc-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.416299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416278 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:37.416299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416285 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-bin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-multus\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416332 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-systemd-units\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416359 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-tmp-dir\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-multus-certs\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416462 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-kubelet-config\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-var-lib-kubelet\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-hosts-file\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.416584 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416563 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-conf-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-etc-kubernetes\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416669 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-systemd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416692 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysconfig\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416717 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-systemd\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-hostroot\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxw25\" (UniqueName: \"kubernetes.io/projected/fdee392a-98f2-4018-ba57-23a7bc2952b7-kube-api-access-rxw25\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3206ad3-d2a7-4eec-81cb-db3a09dee305-serviceca\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-lib-modules\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416897 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-tmp-dir\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416945 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-var-lib-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.416988 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416977 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-multus-certs\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417020 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-etc-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417162 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-multus\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417215 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-systemd-units\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417256 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.417303 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417320 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-kubelet-config\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.417378 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret podName:dc38ff14-bc2b-49aa-b37e-a32248814b35 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.917353211 +0000 UTC m=+3.072859127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret") pod "global-pull-secret-syncer-ndjmx" (UID: "dc38ff14-bc2b-49aa-b37e-a32248814b35") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417373 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-var-lib-kubelet\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.417465 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417429 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.417823 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417479 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-hosts-file\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.417823 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417564 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-hostroot\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.417823 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417604 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-conf-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.417823 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417641 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-etc-kubernetes\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.417823 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417681 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-systemd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.417823 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417689 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.417823 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417738 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-systemd\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.417823 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417741 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysconfig\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417834 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.416977 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gwq\" (UniqueName: \"kubernetes.io/projected/f9a86134-c854-474c-8a22-8d9732ee781a-kube-api-access-c5gwq\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-config\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417948 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-lib-modules\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417963 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-script-lib\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.417993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrs9\" (UniqueName: \"kubernetes.io/projected/a3206ad3-d2a7-4eec-81cb-db3a09dee305-kube-api-access-fsrs9\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-modprobe-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418048 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-etc-selinux\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418075 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-k8s-cni-cncf-io\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418124 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-log-socket\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418166 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfc9n\" (UniqueName: \"kubernetes.io/projected/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-kube-api-access-hfc9n\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-dbus\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-cni-binary-copy\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418224 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418239 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3206ad3-d2a7-4eec-81cb-db3a09dee305-serviceca\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3206ad3-d2a7-4eec-81cb-db3a09dee305-host\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-binary-copy\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-konnectivity-ca\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418321 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418397 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-log-socket\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418496 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-config\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418615 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-modprobe-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418649 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-dbus\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418774 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-script-lib\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-node-log\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovn-node-metrics-cert\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d3607406-fa52-4ff4-a4e3-9a41a050d66e-iptables-alerter-script\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418902 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3206ad3-d2a7-4eec-81cb-db3a09dee305-host\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418931 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418953 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-socket-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-kubelet\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.418994 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.418997 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-node-log\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-env-overrides\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419068 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-system-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419090 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-etc-selinux\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419093 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-netns\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419130 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-kubelet\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3607406-fa52-4ff4-a4e3-9a41a050d66e-host-slash\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-tuned\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-sys-fs\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419227 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-cni-binary-copy\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419235 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g724w\" (UniqueName: \"kubernetes.io/projected/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-kube-api-access-g724w\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-cnibin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-netd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419335 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-kubernetes\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419359 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-tmp\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-slash\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.419606 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-agent-certs\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-conf\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419488 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-sys\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419499 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-kubelet\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419514 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbp5h\" (UniqueName: \"kubernetes.io/projected/2063b915-4bd7-424a-a9d1-8d8065c79166-kube-api-access-lbp5h\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419543 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-netns\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419570 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-ovn\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-bin\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cnibin\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419629 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d3607406-fa52-4ff4-a4e3-9a41a050d66e-iptables-alerter-script\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419673 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-host\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419699 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-socket-dir-parent\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.419708 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419728 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znb5\" (UniqueName: \"kubernetes.io/projected/d3607406-fa52-4ff4-a4e3-9a41a050d66e-kube-api-access-6znb5\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.419760 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.919743145 +0000 UTC m=+3.075249057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:37.420423 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-run\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419796 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419818 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xk87\" (UniqueName: \"kubernetes.io/projected/75eff6b2-9975-4809-a676-f78efa1d45fa-kube-api-access-6xk87\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-registration-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-device-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419912 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-daemon-config\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420025 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3607406-fa52-4ff4-a4e3-9a41a050d66e-host-slash\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420214 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-system-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420429 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-daemon-config\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419463 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-k8s-cni-cncf-io\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420499 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-run\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420574 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-slash\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420619 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-netd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420632 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-sys-fs\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420668 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-cnibin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.421156 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421031 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-sys\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421304 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-registration-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421309 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-kubelet\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421335 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-conf\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.420578 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-device-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421451 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-kubernetes\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421571 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.419132 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-netns\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421634 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-bin\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421662 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-netns\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421715 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-ovn\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421746 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-binary-copy\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421822 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-host\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.421827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421830 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cnibin\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.422482 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421905 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-socket-dir-parent\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.422482 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.421997 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-konnectivity-ca\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:37.422482 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.422170 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.422482 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.422345 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-env-overrides\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.422482 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.422451 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.422482 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.422466 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-socket-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.424383 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.424327 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-tmp\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.424609 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.424589 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovn-node-metrics-cert\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.424703 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.424622 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-agent-certs\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:37.424703 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.424688 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-tuned\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.443884 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.443864 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:37.443884 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.443886 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:37.444066 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.443897 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rshc4 for pod openshift-network-diagnostics/network-check-target-tqb8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:37.444066 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.443953 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4 podName:1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:37.943934569 +0000 UTC m=+3.099440528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rshc4" (UniqueName: "kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4") pod "network-check-target-tqb8d" (UID: "1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:37.447528 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.447330 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xk87\" (UniqueName: \"kubernetes.io/projected/75eff6b2-9975-4809-a676-f78efa1d45fa-kube-api-access-6xk87\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.447528 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.447470 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znb5\" (UniqueName: \"kubernetes.io/projected/d3607406-fa52-4ff4-a4e3-9a41a050d66e-kube-api-access-6znb5\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.448642 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.448611 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g724w\" (UniqueName: \"kubernetes.io/projected/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-kube-api-access-g724w\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.450769 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.450719 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrs9\" (UniqueName: \"kubernetes.io/projected/a3206ad3-d2a7-4eec-81cb-db3a09dee305-kube-api-access-fsrs9\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.451826 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.451783 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gwq\" (UniqueName: \"kubernetes.io/projected/f9a86134-c854-474c-8a22-8d9732ee781a-kube-api-access-c5gwq\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.452603 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.452577 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscx7\" (UniqueName: \"kubernetes.io/projected/60f37ff5-b4ee-46b6-a28c-7db9091eef76-kube-api-access-bscx7\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.455154 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.455132 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfc9n\" (UniqueName: \"kubernetes.io/projected/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-kube-api-access-hfc9n\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:37.456713 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.456690 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxw25\" (UniqueName: \"kubernetes.io/projected/fdee392a-98f2-4018-ba57-23a7bc2952b7-kube-api-access-rxw25\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.460696 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.460669 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbp5h\" (UniqueName: \"kubernetes.io/projected/2063b915-4bd7-424a-a9d1-8d8065c79166-kube-api-access-lbp5h\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.543432 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.543400 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:56:37.548384 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.548359 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:56:37.554494 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:37.554381 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a86134_c854_474c_8a22_8d9732ee781a.slice/crio-5bb2e97ba6e566df65f539d64efbae58980c43ff99dabe163fe6d6b7e0c34b66 WatchSource:0}: Error finding container 5bb2e97ba6e566df65f539d64efbae58980c43ff99dabe163fe6d6b7e0c34b66: Status 404 returned error can't find the container with id 5bb2e97ba6e566df65f539d64efbae58980c43ff99dabe163fe6d6b7e0c34b66 Nov 02 13:56:37.559912 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.559890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w8qtw" Nov 02 13:56:37.566541 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.566523 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:56:37.589605 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:37.589577 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc568d04d_8bb4_41bc_ad89_ba2f09a1c3e5.slice/crio-8f95bbd6e57e271c74bd542a7df7a8b721dcc5d8c584f895bdd2d0df2efd6b48 WatchSource:0}: Error finding container 8f95bbd6e57e271c74bd542a7df7a8b721dcc5d8c584f895bdd2d0df2efd6b48: Status 404 returned error can't find the container with id 8f95bbd6e57e271c74bd542a7df7a8b721dcc5d8c584f895bdd2d0df2efd6b48 Nov 02 13:56:37.593883 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.593820 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:56:37.604024 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.603999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:56:37.604746 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:37.604725 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f37ff5_b4ee_46b6_a28c_7db9091eef76.slice/crio-a917ec2b852414f3d63b43e512bf05f1525b306efe2da1c44400180c3e43c2c4 WatchSource:0}: Error finding container a917ec2b852414f3d63b43e512bf05f1525b306efe2da1c44400180c3e43c2c4: Status 404 returned error can't find the container with id a917ec2b852414f3d63b43e512bf05f1525b306efe2da1c44400180c3e43c2c4 Nov 02 13:56:37.606755 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.606737 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:37.614579 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.614559 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:37.616640 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.616618 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:56:37.618139 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:37.618114 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3607406_fa52_4ff4_a4e3_9a41a050d66e.slice/crio-8c8436bc89fdc848a39d3b2c858f0a1f53f0a76650cfd5f9bcc55cbbe9387487 WatchSource:0}: Error finding container 8c8436bc89fdc848a39d3b2c858f0a1f53f0a76650cfd5f9bcc55cbbe9387487: Status 404 returned error can't find the container with id 8c8436bc89fdc848a39d3b2c858f0a1f53f0a76650cfd5f9bcc55cbbe9387487 Nov 02 13:56:37.620295 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:37.620270 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a1d83b_52ea_4fcd_9843_4eb8199d3d65.slice/crio-2eb0038f417faf0433762faaeee6c659c4cec021320bfe707db383d549a0208c WatchSource:0}: Error finding container 2eb0038f417faf0433762faaeee6c659c4cec021320bfe707db383d549a0208c: Status 404 returned error can't find the container with id 2eb0038f417faf0433762faaeee6c659c4cec021320bfe707db383d549a0208c Nov 02 13:56:37.628057 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:37.628032 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdee392a_98f2_4018_ba57_23a7bc2952b7.slice/crio-89e09af58061a174f016db6ce36d2b96d1bb32a5be79ee2916d0782ed61bb5f0 WatchSource:0}: Error finding container 89e09af58061a174f016db6ce36d2b96d1bb32a5be79ee2916d0782ed61bb5f0: Status 404 returned error can't find the container with id 89e09af58061a174f016db6ce36d2b96d1bb32a5be79ee2916d0782ed61bb5f0 Nov 02 13:56:37.631879 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:56:37.631856 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eff6b2_9975_4809_a676_f78efa1d45fa.slice/crio-621ebd450a9d8c6b57cf7c9f1d075f35e6c6c702f98d1c3fba1397ae79e3a563 WatchSource:0}: Error finding container 621ebd450a9d8c6b57cf7c9f1d075f35e6c6c702f98d1c3fba1397ae79e3a563: Status 404 returned error can't find the container with id 621ebd450a9d8c6b57cf7c9f1d075f35e6c6c702f98d1c3fba1397ae79e3a563 Nov 02 13:56:37.922429 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.922283 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:37.922528 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.922429 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:37.922528 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:37.922476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:37.922528 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.922487 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret podName:dc38ff14-bc2b-49aa-b37e-a32248814b35 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:38.922467961 +0000 UTC m=+4.077973880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret") pod "global-pull-secret-syncer-ndjmx" (UID: "dc38ff14-bc2b-49aa-b37e-a32248814b35") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:37.922670 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.922540 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:37.922670 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:37.922580 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:38.92256633 +0000 UTC m=+4.078072238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:38.022978 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.022892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:38.023213 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:38.023087 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:38.023213 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:38.023116 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:38.023213 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:38.023128 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rshc4 for pod openshift-network-diagnostics/network-check-target-tqb8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:38.023213 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:38.023192 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4 podName:1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:39.023172457 +0000 UTC m=+4.178678358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshc4" (UniqueName: "kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4") pod "network-check-target-tqb8d" (UID: "1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:38.267323 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.267176 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-03-30 06:09:35.688289935 +0000 UTC" Nov 02 13:56:38.267323 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.267243 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12328h12m57.421050834s" Nov 02 13:56:38.276268 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.276132 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" event={"ID":"75eff6b2-9975-4809-a676-f78efa1d45fa","Type":"ContainerStarted","Data":"621ebd450a9d8c6b57cf7c9f1d075f35e6c6c702f98d1c3fba1397ae79e3a563"} Nov 02 13:56:38.279492 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.279438 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"89e09af58061a174f016db6ce36d2b96d1bb32a5be79ee2916d0782ed61bb5f0"} Nov 02 13:56:38.289083 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.289020 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w8qtw" event={"ID":"2063b915-4bd7-424a-a9d1-8d8065c79166","Type":"ContainerStarted","Data":"35e27cab57070c454541a79e8e62089fb6cbd29c3fffc9bab91083f608a872e8"} Nov 02 13:56:38.300586 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.300511 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kjcwb" event={"ID":"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5","Type":"ContainerStarted","Data":"8f95bbd6e57e271c74bd542a7df7a8b721dcc5d8c584f895bdd2d0df2efd6b48"} Nov 02 13:56:38.317662 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.317636 2583 generic.go:356] "Generic (PLEG): container finished" podID="04b7d7036f47f423ed3f88aa11798947" containerID="ff23928bbd43372e67561800bc60ab15d29720d97ad3b623551bccfad8653a49" exitCode=0 Nov 02 13:56:38.317817 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.317715 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"04b7d7036f47f423ed3f88aa11798947","Type":"ContainerDied","Data":"ff23928bbd43372e67561800bc60ab15d29720d97ad3b623551bccfad8653a49"} Nov 02 13:56:38.332775 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.332750 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" event={"ID":"6e4f9ec4cb1fc6480c545594116d855c","Type":"ContainerStarted","Data":"31ee5d4ca175d67b4bc46b94a91ee22b1b377175c9b9b6ee829cc77a0f7624ff"} Nov 02 13:56:38.339615 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.339557 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5q6hs" event={"ID":"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65","Type":"ContainerStarted","Data":"2eb0038f417faf0433762faaeee6c659c4cec021320bfe707db383d549a0208c"} Nov 02 13:56:38.357320 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.357296 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7fcg" event={"ID":"d3607406-fa52-4ff4-a4e3-9a41a050d66e","Type":"ContainerStarted","Data":"8c8436bc89fdc848a39d3b2c858f0a1f53f0a76650cfd5f9bcc55cbbe9387487"} Nov 02 13:56:38.369194 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.369169 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerStarted","Data":"a917ec2b852414f3d63b43e512bf05f1525b306efe2da1c44400180c3e43c2c4"} Nov 02 13:56:38.377999 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.377975 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dtc82" event={"ID":"a3206ad3-d2a7-4eec-81cb-db3a09dee305","Type":"ContainerStarted","Data":"83fd3630a95e455a63705939ac6f3566d3a1f80dca4913c6742fe8604e6519eb"} Nov 02 13:56:38.381767 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.381740 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" event={"ID":"f9a86134-c854-474c-8a22-8d9732ee781a","Type":"ContainerStarted","Data":"5bb2e97ba6e566df65f539d64efbae58980c43ff99dabe163fe6d6b7e0c34b66"} Nov 02 13:56:38.930265 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.930226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:38.930466 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:38.930315 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:38.930535 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:38.930475 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:38.930592 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:38.930552 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret podName:dc38ff14-bc2b-49aa-b37e-a32248814b35 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:40.930532964 +0000 UTC m=+6.086038867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret") pod "global-pull-secret-syncer-ndjmx" (UID: "dc38ff14-bc2b-49aa-b37e-a32248814b35") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:38.931052 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:38.931018 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:38.931157 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:38.931081 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:40.931065997 +0000 UTC m=+6.086571903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:39.040533 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:39.037979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:39.040533 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:39.038196 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:39.040533 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:39.038223 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:39.040533 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:39.038237 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rshc4 for pod openshift-network-diagnostics/network-check-target-tqb8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:39.040533 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:39.038306 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4 podName:1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:41.038286515 +0000 UTC m=+6.193792446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshc4" (UniqueName: "kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4") pod "network-check-target-tqb8d" (UID: "1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:39.269772 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:39.269701 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:39.270268 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:39.269840 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:39.270268 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:39.270233 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:39.270382 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:39.270320 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:39.270431 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:39.270382 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:39.270479 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:39.270464 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:39.271326 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:39.270555 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-06-30 00:18:00.661888534 +0000 UTC" Nov 02 13:56:39.271326 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:39.270607 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14530h21m21.391285881s" Nov 02 13:56:39.393962 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:39.393928 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"04b7d7036f47f423ed3f88aa11798947","Type":"ContainerStarted","Data":"15726d02f4f55e280d3d7330bf6c7bda1eec99175775f1979a08832afb3f2b10"} Nov 02 13:56:39.439902 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:39.439408 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" podStartSLOduration=3.439396 podStartE2EDuration="3.439396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:38.40538979 +0000 UTC m=+3.560895716" watchObservedRunningTime="2026-11-02 13:56:39.439396 +0000 UTC m=+4.594901959" Nov 02 13:56:40.406358 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:40.402247 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerStarted","Data":"8b538b719ddadfe1323ee3d1218728dc206a8a27f32c05fadb3f4bfcc8ca7359"} Nov 02 13:56:40.481647 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:40.481448 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" podStartSLOduration=4.4814283249999995 podStartE2EDuration="4.481428325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:39.439873627 +0000 UTC m=+4.595379550" watchObservedRunningTime="2026-11-02 13:56:40.481428325 +0000 UTC m=+5.636934248" Nov 02 13:56:40.953700 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:40.953660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:40.953908 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:40.953736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:40.953908 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:40.953896 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:40.954016 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:40.953953 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:44.953934257 +0000 UTC m=+10.109440156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:40.954455 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:40.954410 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:40.954550 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:40.954465 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret podName:dc38ff14-bc2b-49aa-b37e-a32248814b35 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:44.954449185 +0000 UTC m=+10.109955103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret") pod "global-pull-secret-syncer-ndjmx" (UID: "dc38ff14-bc2b-49aa-b37e-a32248814b35") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:41.055075 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:41.055040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:41.055260 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:41.055241 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:41.055339 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:41.055267 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:41.055339 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:41.055280 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rshc4 for pod openshift-network-diagnostics/network-check-target-tqb8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:41.055339 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:41.055333 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4 podName:1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:45.055314415 +0000 UTC m=+10.210820328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshc4" (UniqueName: "kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4") pod "network-check-target-tqb8d" (UID: "1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:41.268976 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:41.268909 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:41.269119 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:41.268918 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:41.269119 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:41.269034 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:41.269230 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:41.269139 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:41.269230 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:41.268918 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:41.269315 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:41.269244 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:41.407069 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:41.406258 2583 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="8b538b719ddadfe1323ee3d1218728dc206a8a27f32c05fadb3f4bfcc8ca7359" exitCode=0 Nov 02 13:56:41.407069 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:41.406305 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerDied","Data":"8b538b719ddadfe1323ee3d1218728dc206a8a27f32c05fadb3f4bfcc8ca7359"} Nov 02 13:56:42.411078 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:42.411043 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5q6hs" event={"ID":"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65","Type":"ContainerStarted","Data":"323abd404f986a4394ed0689e08a42a28747c5919b012f8d5aa877662e54ee36"} Nov 02 13:56:43.269024 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:43.268944 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:43.269024 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:43.269008 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:43.269224 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:43.269018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:43.269224 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:43.269121 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:43.269572 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:43.269547 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:43.269674 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:43.269629 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:43.415931 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:43.415884 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dtc82" event={"ID":"a3206ad3-d2a7-4eec-81cb-db3a09dee305","Type":"ContainerStarted","Data":"2de836867ddecad925c3225e4be42a19276fb3aa6af92000a62f0044998406fb"} Nov 02 13:56:43.422123 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:43.422059 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" event={"ID":"f9a86134-c854-474c-8a22-8d9732ee781a","Type":"ContainerStarted","Data":"234c020afde08669894050cf3dcb5a6e16f9fbfea4c5f48d3533c47ea23651d9"} Nov 02 13:56:43.459564 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:43.459505 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5q6hs" podStartSLOduration=4.702021387 podStartE2EDuration="8.459487827s" totalImagesPullingTime="3.75746644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:42.473410843 +0000 UTC m=+7.628916767" watchObservedRunningTime="2026-11-02 13:56:43.459487827 +0000 UTC m=+8.614993749" Nov 02 13:56:44.615904 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:44.615877 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 13:56:44.988459 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:44.988387 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:44.988597 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:44.988462 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:44.988645 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:44.988594 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:44.988691 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:44.988655 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:52.988635754 +0000 UTC m=+18.144141656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:44.989053 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:44.989035 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:44.989111 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:44.989095 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret podName:dc38ff14-bc2b-49aa-b37e-a32248814b35 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:52.989080626 +0000 UTC m=+18.144586528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret") pod "global-pull-secret-syncer-ndjmx" (UID: "dc38ff14-bc2b-49aa-b37e-a32248814b35") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:45.089381 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.089352 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:45.089533 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:45.089493 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:45.089533 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:45.089507 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:45.089533 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:45.089517 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rshc4 for pod openshift-network-diagnostics/network-check-target-tqb8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:45.089655 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:45.089559 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4 podName:1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73 nodeName:}" failed. No retries permitted until 2026-11-02 13:56:53.089546748 +0000 UTC m=+18.245052647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshc4" (UniqueName: "kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4") pod "network-check-target-tqb8d" (UID: "1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:45.270529 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.270507 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:45.270651 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:45.270613 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:45.270718 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.270657 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:45.270770 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:45.270739 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:45.270837 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.270822 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:45.270910 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:45.270891 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:45.271026 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.270945 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:56:44.615902217Z","UUID":"4120a61b-32d9-42b5-b981-608f1f77d040","Handler":null,"Name":"","Endpoint":""} Nov 02 13:56:45.275967 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.275947 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 13:56:45.276046 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.275979 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 13:56:45.432374 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.432340 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" event={"ID":"f9a86134-c854-474c-8a22-8d9732ee781a","Type":"ContainerStarted","Data":"26f4ebf4f91fa47e82d9a2ccee1d93c78c6a94ed352f796ade664a1fe2541e07"} Nov 02 13:56:45.434834 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.434812 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w8qtw" event={"ID":"2063b915-4bd7-424a-a9d1-8d8065c79166","Type":"ContainerStarted","Data":"3af4ee7ad3eda6bd5deb505a0f9be84eebc320a97412dc490783cc0a35809f51"} Nov 02 13:56:45.439985 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.439958 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kjcwb" event={"ID":"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5","Type":"ContainerStarted","Data":"8fac1b2a56a385e5f17cc6c05c75e5cf8c9bce019f15c920981c16c333a6528f"} Nov 02 13:56:45.455786 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.455751 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:45.456571 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.456554 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:45.481630 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.481593 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-w8qtw" podStartSLOduration=3.345150347 podStartE2EDuration="10.48157895s" totalImagesPullingTime="7.136428603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:45.481377458 +0000 UTC m=+10.636883380" watchObservedRunningTime="2026-11-02 13:56:45.48157895 +0000 UTC m=+10.637084874" Nov 02 13:56:45.481747 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.481705 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dtc82" podStartSLOduration=5.694924533 podStartE2EDuration="10.481699053s" totalImagesPullingTime="4.78677452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:43.460436972 +0000 UTC m=+8.615942894" watchObservedRunningTime="2026-11-02 13:56:45.481699053 +0000 UTC m=+10.637204976" Nov 02 13:56:45.528710 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:45.528481 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kjcwb" podStartSLOduration=2.890972375 podStartE2EDuration="10.528465915s" totalImagesPullingTime="7.63749354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:45.527931382 +0000 UTC m=+10.683437304" watchObservedRunningTime="2026-11-02 13:56:45.528465915 +0000 UTC m=+10.683971837" Nov 02 13:56:46.444299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:46.444261 2583 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="20a3052ad233be5429eb9a36657987a0046bef8c279478a021ec4539b5df6a61" exitCode=0 Nov 02 13:56:46.444761 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:46.444346 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerDied","Data":"20a3052ad233be5429eb9a36657987a0046bef8c279478a021ec4539b5df6a61"} Nov 02 13:56:46.448812 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:46.448770 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" event={"ID":"f9a86134-c854-474c-8a22-8d9732ee781a","Type":"ContainerStarted","Data":"b1440ede935702b0c9f86ca59769ce681cac49fc73b7b7b4b54ec6b35a8432cf"} Nov 02 13:56:46.554376 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:46.554330 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" podStartSLOduration=3.833741064 podStartE2EDuration="11.554314104s" totalImagesPullingTime="7.72057304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:46.554239155 +0000 UTC m=+11.709745075" watchObservedRunningTime="2026-11-02 13:56:46.554314104 +0000 UTC m=+11.709820022" Nov 02 13:56:47.267109 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.267039 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:47.267109 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.267055 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:47.267267 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:47.267207 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:47.267371 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.267336 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:47.267442 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:47.267422 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:47.267542 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:47.267521 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:47.452086 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.452055 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7fcg" event={"ID":"d3607406-fa52-4ff4-a4e3-9a41a050d66e","Type":"ContainerStarted","Data":"09febf1c3ec0d5e18afdfeab8364d4815fc351273b098cf44134be13210627b4"} Nov 02 13:56:47.454336 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.454308 2583 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="f2547009e6a1659c3ce8215765edf1b3156bfb1d46f990278452772b103cb0fc" exitCode=0 Nov 02 13:56:47.454446 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.454355 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerDied","Data":"f2547009e6a1659c3ce8215765edf1b3156bfb1d46f990278452772b103cb0fc"} Nov 02 13:56:47.455967 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.455942 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" event={"ID":"75eff6b2-9975-4809-a676-f78efa1d45fa","Type":"ContainerStarted","Data":"890493dd74f52f3636eaea0555857d356065c2d4abe1b37ac4696db4c325eac5"} Nov 02 13:56:47.497108 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.497062 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r7fcg" podStartSLOduration=4.799478838 podStartE2EDuration="12.49704756s" totalImagesPullingTime="7.697568722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:47.496414262 +0000 UTC m=+12.651920185" watchObservedRunningTime="2026-11-02 13:56:47.49704756 +0000 UTC m=+12.652553484" Nov 02 13:56:47.609235 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:47.609146 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" podStartSLOduration=3.174343556 podStartE2EDuration="12.609132444s" totalImagesPullingTime="9.434788888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:47.608395805 +0000 UTC m=+12.763901725" watchObservedRunningTime="2026-11-02 13:56:47.609132444 +0000 UTC m=+12.764638365" Nov 02 13:56:48.460881 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:48.460793 2583 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="521454d97ea5cec9fd548d8a725f8372dc2dcc68ce347ef2cc36d47a2461cea4" exitCode=0 Nov 02 13:56:48.461344 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:48.460877 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerDied","Data":"521454d97ea5cec9fd548d8a725f8372dc2dcc68ce347ef2cc36d47a2461cea4"} Nov 02 13:56:49.269112 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:49.268881 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:49.269299 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:49.269209 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:49.269299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:49.268893 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:49.269299 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:49.268922 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:49.269299 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:49.269290 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:49.269439 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:49.269369 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:50.468764 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:50.468732 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerStarted","Data":"74384ba9ba4046c6f3c0c912da35d059f21b388b4613d7e634b37ba9344e62c9"} Nov 02 13:56:51.267229 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:51.267196 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:51.267229 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:51.267209 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:51.267429 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:51.267338 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:51.267622 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:51.267606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:51.267720 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:51.267689 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:51.267759 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:51.267746 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:51.473080 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:51.473047 2583 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="74384ba9ba4046c6f3c0c912da35d059f21b388b4613d7e634b37ba9344e62c9" exitCode=0 Nov 02 13:56:51.473476 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:51.473094 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerDied","Data":"74384ba9ba4046c6f3c0c912da35d059f21b388b4613d7e634b37ba9344e62c9"} Nov 02 13:56:51.877787 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:51.877724 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:51.877902 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:51.877857 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:56:51.878495 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:51.878477 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:56:52.478132 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:52.477957 2583 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="9c273344a6a9bfc649934977cedc20689d2b6685a680845be88a9eb20394a630" exitCode=0 Nov 02 13:56:52.478642 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:52.478034 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerDied","Data":"9c273344a6a9bfc649934977cedc20689d2b6685a680845be88a9eb20394a630"} Nov 02 13:56:52.481218 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:52.481196 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"b9435317918316fee10b473caa9784790333a07f228b0b40aabc82149a83adab"} Nov 02 13:56:52.481287 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:52.481258 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"a9a86b196fd41a86cffb0d837e4a8c4c03881361ff6ce96331ef2b4d749a2acb"} Nov 02 13:56:52.481326 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:52.481292 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"12d494850873b74e9815b3603e70146f5ba1876303bafe9002864977e9b40460"} Nov 02 13:56:52.481326 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:52.481321 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"9185c0a6f8f41973348a981a7033b10680a1a6b4289c3c6d4a3a4729882499aa"} Nov 02 13:56:52.481389 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:52.481347 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"36b5a2a9cbdd3f03f490e70e1cf4b99ef3a3f4e01e6468ff5e3fc52b08104b82"} Nov 02 13:56:52.481389 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:52.481375 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"77ff3215e9c8554ac7108bdf3767d251ae1d6bb16e45484710f963a485d2add8"} Nov 02 13:56:53.049429 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:53.049401 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:53.049551 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:53.049442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:53.049551 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.049537 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:53.049628 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.049588 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:53.049628 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.049600 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret podName:dc38ff14-bc2b-49aa-b37e-a32248814b35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:09.049582091 +0000 UTC m=+34.205087993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret") pod "global-pull-secret-syncer-ndjmx" (UID: "dc38ff14-bc2b-49aa-b37e-a32248814b35") : object "kube-system"/"original-pull-secret" not registered Nov 02 13:56:53.049628 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.049627 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:09.049615437 +0000 UTC m=+34.205121339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 13:56:53.150405 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:53.150377 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:53.150531 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.150513 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 13:56:53.150581 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.150534 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 13:56:53.150581 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.150544 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rshc4 for pod openshift-network-diagnostics/network-check-target-tqb8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:53.150641 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.150591 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4 podName:1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:09.150575725 +0000 UTC m=+34.306081626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rshc4" (UniqueName: "kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4") pod "network-check-target-tqb8d" (UID: "1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 13:56:53.267375 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:53.267354 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:53.267466 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.267446 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:53.267523 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:53.267504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:53.267626 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.267608 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:53.267674 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:53.267662 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:53.267728 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:53.267715 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:53.487562 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:53.487535 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" event={"ID":"60f37ff5-b4ee-46b6-a28c-7db9091eef76","Type":"ContainerStarted","Data":"a6b12e39ec41582e013022d9b70fd68003ebf7dd4c55f3dcd57d5d6673259dce"} Nov 02 13:56:54.492484 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:54.492458 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"26795f85ff523b78d620eeac28796a32da2bd3fb2af5d06ab2916f7da0ba5544"} Nov 02 13:56:55.267078 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:55.267044 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:55.267078 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:55.267072 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:55.267310 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:55.267146 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:55.267310 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:55.267155 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:55.267310 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:55.267215 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:55.267426 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:55.267323 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:57.267465 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.267429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:57.268138 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.267434 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:57.268138 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:57.267527 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:57.268138 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:57.267633 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:57.268138 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.267443 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:57.268138 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:57.267741 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:57.501703 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.501674 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" event={"ID":"fdee392a-98f2-4018-ba57-23a7bc2952b7","Type":"ContainerStarted","Data":"af4b9c7a4750dc5da9c5866d75bd9a512cf5484a54c362c7708e74ea16cb9bf9"} Nov 02 13:56:57.502034 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.502002 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:57.502185 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.502081 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:57.517566 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.517511 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:57.517716 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.517702 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:56:57.598477 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.598436 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" podStartSLOduration=8.174460648 podStartE2EDuration="22.598422769s" totalImagesPullingTime="14.423962121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:57.598341152 +0000 UTC m=+22.753847074" watchObservedRunningTime="2026-11-02 13:56:57.598422769 +0000 UTC m=+22.753928689" Nov 02 13:56:57.599144 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:57.599082 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wpzw4" podStartSLOduration=12.790516942 podStartE2EDuration="22.599073497s" totalImagesPullingTime="9.808556555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:53.54055169 +0000 UTC m=+18.696057609" watchObservedRunningTime="2026-11-02 13:56:57.599073497 +0000 UTC m=+22.754579419" Nov 02 13:56:58.504280 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:58.504255 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:56:58.558605 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:58.558569 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqb8d"] Nov 02 13:56:58.558743 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:58.558729 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:56:58.558868 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:58.558850 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:56:58.564392 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:58.564365 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ndjmx"] Nov 02 13:56:58.564474 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:58.564447 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:56:58.564532 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:58.564517 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:56:58.571743 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:58.571724 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bjbp9"] Nov 02 13:56:58.571825 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:58.571818 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:56:58.571912 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:56:58.571887 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:56:58.741379 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:56:58.741294 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:00.267521 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:00.267488 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:00.267521 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:00.267511 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:00.268048 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:00.267488 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:00.268048 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:00.267592 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:57:00.268048 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:00.267669 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:57:00.268048 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:00.267756 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:57:02.266884 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:02.266849 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:02.267348 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:02.266925 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:02.267348 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:02.267007 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:02.267348 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:02.267019 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:57:02.267348 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:02.267080 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:57:02.267348 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:02.267151 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:57:04.267510 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.267485 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:04.267841 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.267517 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:04.267841 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.267517 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:04.267841 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:04.267582 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bjbp9" podUID="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" Nov 02 13:57:04.267841 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:04.267679 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndjmx" podUID="dc38ff14-bc2b-49aa-b37e-a32248814b35" Nov 02 13:57:04.267841 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:04.267751 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tqb8d" podUID="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" Nov 02 13:57:04.592754 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.592734 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeReady" Nov 02 13:57:04.592893 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.592865 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 13:57:04.654650 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.654625 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bv8r5"] Nov 02 13:57:04.659489 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.659467 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4q2dh"] Nov 02 13:57:04.659588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.659565 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.662583 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.662563 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:04.682874 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.682136 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 13:57:04.682874 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.682286 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 13:57:04.682874 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.682317 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 13:57:04.682874 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.682471 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 13:57:04.684331 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.684219 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bv8r5"] Nov 02 13:57:04.685330 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.685308 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4q2dh"] Nov 02 13:57:04.692482 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.692459 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 13:57:04.692565 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.692500 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9cgsx\"" Nov 02 13:57:04.692601 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.692593 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9rw8\"" Nov 02 13:57:04.743024 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.743000 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgv6d\" (UniqueName: \"kubernetes.io/projected/b9d5b255-ab45-4fce-9226-a02f7ffec197-kube-api-access-tgv6d\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:04.743120 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.743047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.743120 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.743074 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef68447e-3ad3-4dd2-b353-4aa00584d583-tmp-dir\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.743185 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.743146 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9d5b255-ab45-4fce-9226-a02f7ffec197-cert\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:04.743217 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.743188 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68447e-3ad3-4dd2-b353-4aa00584d583-config-volume\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.743217 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.743205 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxz5z\" (UniqueName: \"kubernetes.io/projected/ef68447e-3ad3-4dd2-b353-4aa00584d583-kube-api-access-bxz5z\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.843712 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.843646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68447e-3ad3-4dd2-b353-4aa00584d583-config-volume\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.843712 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.843679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxz5z\" (UniqueName: \"kubernetes.io/projected/ef68447e-3ad3-4dd2-b353-4aa00584d583-kube-api-access-bxz5z\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.843712 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.843700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgv6d\" (UniqueName: \"kubernetes.io/projected/b9d5b255-ab45-4fce-9226-a02f7ffec197-kube-api-access-tgv6d\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:04.844008 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.843730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.844008 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:04.843832 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:04.844008 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.843857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef68447e-3ad3-4dd2-b353-4aa00584d583-tmp-dir\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.844008 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:04.843888 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:05.343871644 +0000 UTC m=+30.499377560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:04.844008 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.843922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9d5b255-ab45-4fce-9226-a02f7ffec197-cert\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:04.844268 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.844153 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef68447e-3ad3-4dd2-b353-4aa00584d583-tmp-dir\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.844342 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.844321 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68447e-3ad3-4dd2-b353-4aa00584d583-config-volume\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.847473 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.847451 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9d5b255-ab45-4fce-9226-a02f7ffec197-cert\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:04.869332 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.869310 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxz5z\" (UniqueName: \"kubernetes.io/projected/ef68447e-3ad3-4dd2-b353-4aa00584d583-kube-api-access-bxz5z\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:04.869332 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.869322 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgv6d\" (UniqueName: \"kubernetes.io/projected/b9d5b255-ab45-4fce-9226-a02f7ffec197-kube-api-access-tgv6d\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:04.977308 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:04.977279 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:05.290125 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:05.290098 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4q2dh"] Nov 02 13:57:05.293082 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:57:05.293060 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d5b255_ab45_4fce_9226_a02f7ffec197.slice/crio-9a0ab90615e9c97e654a386737c0e438a79f5e4d6c7ae6dfd4f6a8b783452ad4 WatchSource:0}: Error finding container 9a0ab90615e9c97e654a386737c0e438a79f5e4d6c7ae6dfd4f6a8b783452ad4: Status 404 returned error can't find the container with id 9a0ab90615e9c97e654a386737c0e438a79f5e4d6c7ae6dfd4f6a8b783452ad4 Nov 02 13:57:05.348970 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:05.348952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:05.349060 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:05.349048 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:05.349105 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:05.349092 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:06.349079983 +0000 UTC m=+31.504585886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:05.521453 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:05.521429 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4q2dh" event={"ID":"b9d5b255-ab45-4fce-9226-a02f7ffec197","Type":"ContainerStarted","Data":"9a0ab90615e9c97e654a386737c0e438a79f5e4d6c7ae6dfd4f6a8b783452ad4"} Nov 02 13:57:06.267255 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.267221 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:06.267442 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.267283 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:06.267442 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.267382 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:06.284126 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.284108 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 13:57:06.284238 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.284199 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 13:57:06.297738 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.297718 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 13:57:06.298052 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.297883 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 13:57:06.298052 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.297928 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sczcr\"" Nov 02 13:57:06.298052 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.297968 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8ms52\"" Nov 02 13:57:06.298052 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.297984 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xm2nq\"" Nov 02 13:57:06.355367 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:06.355345 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:06.355471 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:06.355429 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:06.355533 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:06.355476 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:08.355460321 +0000 UTC m=+33.510966221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:08.370827 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:08.370789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:08.371310 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:08.370883 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:08.371310 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:08.370957 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:12.370943028 +0000 UTC m=+37.526448945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:08.530256 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:08.530175 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4q2dh" event={"ID":"b9d5b255-ab45-4fce-9226-a02f7ffec197","Type":"ContainerStarted","Data":"0518c7bd542dd09a36ae682d8ceb60d94ed07ad19b5f3c0c482ae40f00736e1d"} Nov 02 13:57:08.574451 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:08.574403 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4q2dh" podStartSLOduration=1.692936523 podStartE2EDuration="4.574389418s" totalImagesPullingTime="2.881452895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:08.573942962 +0000 UTC m=+33.729448882" watchObservedRunningTime="2026-11-02 13:57:08.574389418 +0000 UTC m=+33.729895341" Nov 02 13:57:09.075157 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.075117 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:09.075359 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.075179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:09.075359 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:09.075305 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:09.075472 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:09.075372 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:41.075353172 +0000 UTC m=+66.230859070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : secret "metrics-daemon-secret" not found Nov 02 13:57:09.077464 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.077441 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:09.175452 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.175423 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:09.177933 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.177911 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:09.310668 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.310641 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:09.358139 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.358114 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:09.638344 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.638314 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tqb8d"] Nov 02 13:57:09.641579 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:57:09.641555 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba2eeed_bf6d_4c29_9505_d8a8f9d53e73.slice/crio-528320711e0f4a7b9bca8b3e0a667450112dc2cc628a3a3f01da08edc94c3686 WatchSource:0}: Error finding container 528320711e0f4a7b9bca8b3e0a667450112dc2cc628a3a3f01da08edc94c3686: Status 404 returned error can't find the container with id 528320711e0f4a7b9bca8b3e0a667450112dc2cc628a3a3f01da08edc94c3686 Nov 02 13:57:09.666647 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:09.666626 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ndjmx"] Nov 02 13:57:09.669119 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:57:09.669099 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc38ff14_bc2b_49aa_b37e_a32248814b35.slice/crio-8ff46100aec67fb942dd93ab86434fa0f4be9685f68d8bb3e73b6d800e4d65b3 WatchSource:0}: Error finding container 8ff46100aec67fb942dd93ab86434fa0f4be9685f68d8bb3e73b6d800e4d65b3: Status 404 returned error can't find the container with id 8ff46100aec67fb942dd93ab86434fa0f4be9685f68d8bb3e73b6d800e4d65b3 Nov 02 13:57:10.334410 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.334373 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vwqt2"] Nov 02 13:57:10.355942 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.355914 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.365619 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.365586 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vwqt2"] Nov 02 13:57:10.381958 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.381938 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 13:57:10.382070 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.381992 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xjjmq\"" Nov 02 13:57:10.420974 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.420877 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 13:57:10.421209 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.421168 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 13:57:10.421283 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.421210 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 13:57:10.484386 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.484360 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm"] Nov 02 13:57:10.484569 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.484545 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.484647 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.484630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nth9n\" (UniqueName: \"kubernetes.io/projected/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-api-access-nth9n\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.484712 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.484678 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/166c406a-eb7f-41ce-9d49-71f3fda95c35-crio-socket\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.484768 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.484719 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.484845 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.484807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/166c406a-eb7f-41ce-9d49-71f3fda95c35-data-volume\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.510272 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.510249 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" Nov 02 13:57:10.510604 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.510579 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm"] Nov 02 13:57:10.537588 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.537336 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ndjmx" event={"ID":"dc38ff14-bc2b-49aa-b37e-a32248814b35","Type":"ContainerStarted","Data":"8ff46100aec67fb942dd93ab86434fa0f4be9685f68d8bb3e73b6d800e4d65b3"} Nov 02 13:57:10.538721 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.538701 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqb8d" event={"ID":"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73","Type":"ContainerStarted","Data":"528320711e0f4a7b9bca8b3e0a667450112dc2cc628a3a3f01da08edc94c3686"} Nov 02 13:57:10.572525 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.572496 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5mnrf\"" Nov 02 13:57:10.585209 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.585148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/166c406a-eb7f-41ce-9d49-71f3fda95c35-crio-socket\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.585209 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.585187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.585357 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.585223 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/166c406a-eb7f-41ce-9d49-71f3fda95c35-data-volume\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.585357 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.585275 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.585357 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:10.585288 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:10.585357 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.585302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nth9n\" (UniqueName: \"kubernetes.io/projected/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-api-access-nth9n\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.585357 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:10.585348 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.085332103 +0000 UTC m=+36.240838019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : configmap "kube-rbac-proxy" not found Nov 02 13:57:10.585601 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.585382 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/166c406a-eb7f-41ce-9d49-71f3fda95c35-crio-socket\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.585601 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:10.585434 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:10.585601 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:10.585502 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:11.085474659 +0000 UTC m=+36.240980572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:10.585871 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.585845 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/166c406a-eb7f-41ce-9d49-71f3fda95c35-data-volume\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.637110 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.637088 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nth9n\" (UniqueName: \"kubernetes.io/projected/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-api-access-nth9n\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:10.643383 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.643345 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:10.643728 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.643398 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 13:57:10.685982 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.685954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbphq\" (UniqueName: \"kubernetes.io/projected/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5-kube-api-access-kbphq\") pod \"migrator-66df5d496b-4xbqm\" (UID: \"6efd880e-a7eb-4c81-b081-f3a2ca09e1c5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" Nov 02 13:57:10.787273 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.787239 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbphq\" (UniqueName: \"kubernetes.io/projected/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5-kube-api-access-kbphq\") pod \"migrator-66df5d496b-4xbqm\" (UID: \"6efd880e-a7eb-4c81-b081-f3a2ca09e1c5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" Nov 02 13:57:10.813728 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.813700 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbphq\" (UniqueName: \"kubernetes.io/projected/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5-kube-api-access-kbphq\") pod \"migrator-66df5d496b-4xbqm\" (UID: \"6efd880e-a7eb-4c81-b081-f3a2ca09e1c5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" Nov 02 13:57:10.835515 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:10.835462 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" Nov 02 13:57:11.091010 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:11.090936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:11.091137 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:11.091011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:11.091137 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:11.091084 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:11.091137 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:11.091131 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:12.091115745 +0000 UTC m=+37.246621645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : configmap "kube-rbac-proxy" not found Nov 02 13:57:11.091479 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:11.091457 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:11.091636 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:11.091512 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:12.091497956 +0000 UTC m=+37.247003870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:11.174453 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:11.174422 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm"] Nov 02 13:57:11.179079 ip-10-0-134-9 kubenswrapper[2583]: W1102 13:57:11.179042 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efd880e_a7eb_4c81_b081_f3a2ca09e1c5.slice/crio-9cb6fb493b45344844ccb77d4c8d68b49dd084ca07519d052513e7531d4cc833 WatchSource:0}: Error finding container 9cb6fb493b45344844ccb77d4c8d68b49dd084ca07519d052513e7531d4cc833: Status 404 returned error can't find the container with id 9cb6fb493b45344844ccb77d4c8d68b49dd084ca07519d052513e7531d4cc833 Nov 02 13:57:11.543018 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:11.542979 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" event={"ID":"6efd880e-a7eb-4c81-b081-f3a2ca09e1c5","Type":"ContainerStarted","Data":"9cb6fb493b45344844ccb77d4c8d68b49dd084ca07519d052513e7531d4cc833"} Nov 02 13:57:12.101106 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:12.101071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:12.101517 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:12.101159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:12.101517 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:12.101243 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:12.101517 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:12.101303 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:14.101285234 +0000 UTC m=+39.256791146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : configmap "kube-rbac-proxy" not found Nov 02 13:57:12.101517 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:12.101243 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:12.101517 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:12.101385 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:14.101363156 +0000 UTC m=+39.256869070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:12.404125 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:12.404084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:12.404283 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:12.404253 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:12.404356 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:12.404315 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:20.404297316 +0000 UTC m=+45.559803230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:13.549491 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:13.549453 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tqb8d" event={"ID":"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73","Type":"ContainerStarted","Data":"7a1d90bcb75e33406a161621f8daf88445967f2055aba2d9b75dabbc41e2019a"} Nov 02 13:57:13.549972 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:13.549521 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:13.551354 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:13.551329 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" event={"ID":"6efd880e-a7eb-4c81-b081-f3a2ca09e1c5","Type":"ContainerStarted","Data":"72a9750d1169c731caa25feefe3299586d1b75958d47d99767e2773a111713c1"} Nov 02 13:57:13.593717 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:13.593624 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tqb8d" podStartSLOduration=34.907594388 podStartE2EDuration="38.593607982s" totalImagesPullingTime="3.686013594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:13.592520125 +0000 UTC m=+38.748026047" watchObservedRunningTime="2026-11-02 13:57:13.593607982 +0000 UTC m=+38.749113906" Nov 02 13:57:13.637887 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:13.637833 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" podStartSLOduration=1.92988934 podStartE2EDuration="3.637817699s" totalImagesPullingTime="1.707928359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:13.63682619 +0000 UTC m=+38.792332110" watchObservedRunningTime="2026-11-02 13:57:13.637817699 +0000 UTC m=+38.793323620" Nov 02 13:57:14.117526 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:14.117486 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:14.117709 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:14.117627 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:14.117709 ip-10-0-134-9 kubenswrapper[2583]: I1102 13:57:14.117674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:14.117709 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:14.117703 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:18.117683202 +0000 UTC m=+43.273189125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : configmap "kube-rbac-proxy" not found Nov 02 13:57:14.117905 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:14.117792 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:14.117905 ip-10-0-134-9 kubenswrapper[2583]: E1102 13:57:14.117848 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:18.11783291 +0000 UTC m=+43.273338832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:14.908354 ip-10-0-134-9 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 13:57:14.922192 ip-10-0-134-9 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 13:57:14.922358 ip-10-0-134-9 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 13:57:14.922405 ip-10-0-134-9 systemd[1]: kubelet.service: Consumed 2.069s CPU time. Nov 02 13:57:14.934234 ip-10-0-134-9 systemd[1]: Starting Kubernetes Kubelet... Nov 02 13:57:14.991242 ip-10-0-134-9 kubenswrapper[4431]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:57:14.991242 ip-10-0-134-9 kubenswrapper[4431]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 13:57:14.991242 ip-10-0-134-9 kubenswrapper[4431]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:57:14.991242 ip-10-0-134-9 kubenswrapper[4431]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 13:57:14.993253 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993189 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993259 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993264 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993267 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993270 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993273 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993276 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993279 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993281 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993284 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993286 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993289 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993292 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993296 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993300 4431 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.993298 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993304 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993307 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993310 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993313 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993332 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993336 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993339 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993343 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993347 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993350 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993352 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993355 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993358 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993361 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993363 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993366 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993368 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993371 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993374 4431 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.993693 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993377 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993380 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993382 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993385 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993387 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993390 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993393 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993395 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993398 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993400 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993403 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993405 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993408 4431 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993411 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993413 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993417 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993419 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993423 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993426 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.994186 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993429 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993431 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993434 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993437 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993440 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993442 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993445 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993448 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993450 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993453 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993456 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993458 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993461 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993465 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993467 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993470 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993473 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993475 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993478 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993481 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.994650 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993484 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993487 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993489 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993492 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993495 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993497 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993500 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993502 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993505 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993508 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993510 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993513 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993516 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993518 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993521 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993523 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993526 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993529 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993531 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993534 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.995150 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993536 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993539 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993542 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993544 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993547 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993550 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993553 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993556 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993558 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993561 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993563 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993566 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.993569 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994288 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994295 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994298 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994300 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994304 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994307 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994310 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:14.995715 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994313 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994315 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994318 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994321 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994323 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994326 4431 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994329 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994331 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994334 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994337 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994339 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994342 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994345 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994347 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994350 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994353 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994355 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994358 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994361 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994364 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:14.996231 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994367 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994369 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994372 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994375 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994377 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994381 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994386 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994389 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994392 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994397 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994399 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994402 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994405 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994408 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994410 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994413 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994416 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994418 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994421 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:14.996716 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994424 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994426 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994429 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994431 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994434 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994437 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994439 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994442 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994445 4431 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994447 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994450 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994452 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994455 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994459 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994463 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994467 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994470 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994472 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994475 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994478 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:14.997189 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994481 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994483 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994487 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994489 4431 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994492 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994495 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994498 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994500 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994502 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994505 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994507 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994510 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994513 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994516 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994518 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994521 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994524 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994527 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994529 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.997881 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994532 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994534 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994537 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994539 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994542 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994544 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994547 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994550 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994552 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994555 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994557 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994559 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994562 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994564 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994567 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994574 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994576 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994579 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994582 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.998656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994584 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.994587 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995063 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995068 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995072 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995074 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995077 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995080 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995082 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995085 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995088 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995091 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995094 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995097 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995099 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995102 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995105 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995107 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995109 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995112 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:14.999140 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995115 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995118 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995121 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995123 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995127 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995131 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995134 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995136 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995140 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995145 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995149 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995152 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995156 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995159 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995162 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995164 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995167 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995173 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995176 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:14.999960 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995179 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995181 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995184 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995187 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995189 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995192 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995194 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995197 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995200 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995202 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995205 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995207 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995210 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995213 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995216 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995218 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995221 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995223 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995226 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:15.000513 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995228 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995231 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995234 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995237 4431 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995239 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995242 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995244 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995247 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995250 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995252 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995255 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995259 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995261 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995265 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995268 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995270 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995273 4431 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995276 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995278 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:15.001098 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995281 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995284 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995286 4431 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995289 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995291 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995294 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995297 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995299 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995302 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995305 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995307 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995310 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995312 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995315 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995318 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995320 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995323 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995325 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995328 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995330 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:15.001575 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995332 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995335 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995338 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995341 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995344 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995347 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995350 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995352 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995355 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995357 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.995360 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995452 4431 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995459 4431 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995468 4431 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995473 4431 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995477 4431 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995481 4431 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995485 4431 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995489 4431 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995492 4431 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995495 4431 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 13:57:15.002082 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995499 4431 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995502 4431 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995505 4431 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995508 4431 flags.go:64] FLAG: --cgroup-root="" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995511 4431 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995514 4431 flags.go:64] FLAG: --client-ca-file="" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995518 4431 flags.go:64] FLAG: --cloud-provider="external" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995521 4431 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995525 4431 flags.go:64] FLAG: --cluster-domain="" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995528 4431 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995531 4431 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995534 4431 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995537 4431 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995541 4431 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995544 4431 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995549 4431 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995552 4431 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995555 4431 flags.go:64] FLAG: --contention-profiling="false" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995558 4431 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995562 4431 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995565 4431 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995568 4431 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995572 4431 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995575 4431 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995578 4431 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 13:57:15.002628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995581 4431 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995584 4431 flags.go:64] FLAG: --enable-server="true" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995587 4431 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995591 4431 flags.go:64] FLAG: --event-burst="100" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995594 4431 flags.go:64] FLAG: --event-qps="50" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995597 4431 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995600 4431 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995603 4431 flags.go:64] FLAG: --eviction-hard="" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995608 4431 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995611 4431 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995614 4431 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995617 4431 flags.go:64] FLAG: --eviction-soft="" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995620 4431 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995625 4431 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995628 4431 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995631 4431 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995634 4431 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995637 4431 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995640 4431 flags.go:64] FLAG: --feature-gates="" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995644 4431 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995647 4431 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995650 4431 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995655 4431 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995659 4431 flags.go:64] FLAG: --healthz-port="10248" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995662 4431 flags.go:64] FLAG: --help="false" Nov 02 13:57:15.003347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995665 4431 flags.go:64] FLAG: --hostname-override="ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995668 4431 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995671 4431 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995674 4431 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995677 4431 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995680 4431 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995683 4431 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995686 4431 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995689 4431 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995692 4431 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995695 4431 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995698 4431 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995701 4431 flags.go:64] FLAG: --kube-reserved="" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995704 4431 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995707 4431 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995710 4431 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995713 4431 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995716 4431 flags.go:64] FLAG: --lock-file="" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995719 4431 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995722 4431 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995727 4431 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995733 4431 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995736 4431 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995738 4431 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 13:57:15.003978 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995741 4431 flags.go:64] FLAG: --logging-format="text" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995744 4431 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995748 4431 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995751 4431 flags.go:64] FLAG: --manifest-url="" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995753 4431 flags.go:64] FLAG: --manifest-url-header="" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995759 4431 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995762 4431 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995765 4431 flags.go:64] FLAG: --max-pods="110" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995768 4431 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995771 4431 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995786 4431 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995789 4431 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995792 4431 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995796 4431 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995798 4431 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995805 4431 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995808 4431 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995811 4431 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995814 4431 flags.go:64] FLAG: --pod-cidr="" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995817 4431 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995820 4431 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995823 4431 flags.go:64] FLAG: --pods-per-core="0" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995826 4431 flags.go:64] FLAG: --port="10250" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995828 4431 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995831 4431 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-063e0fc4fb4fe3492" Nov 02 13:57:15.004551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995835 4431 flags.go:64] FLAG: --qos-reserved="" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995838 4431 flags.go:64] FLAG: --read-only-port="10255" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995841 4431 flags.go:64] FLAG: --register-node="true" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995845 4431 flags.go:64] FLAG: --register-with-taints="" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995849 4431 flags.go:64] FLAG: --registry-burst="10" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995852 4431 flags.go:64] FLAG: --registry-qps="5" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995855 4431 flags.go:64] FLAG: --reserved-cpus="" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995858 4431 flags.go:64] FLAG: --reserved-memory="" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995862 4431 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995865 4431 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995868 4431 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995871 4431 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995874 4431 flags.go:64] FLAG: --runonce="false" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995877 4431 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995880 4431 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995884 4431 flags.go:64] FLAG: --seccomp-default="false" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995886 4431 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995890 4431 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995893 4431 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995896 4431 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995899 4431 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995901 4431 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995904 4431 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995907 4431 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995910 4431 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995913 4431 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 13:57:15.005183 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995916 4431 flags.go:64] FLAG: --system-cgroups="" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995919 4431 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995924 4431 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995927 4431 flags.go:64] FLAG: --tls-cert-file="" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995930 4431 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995940 4431 flags.go:64] FLAG: --tls-min-version="" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995943 4431 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995947 4431 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995950 4431 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995955 4431 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995958 4431 flags.go:64] FLAG: --v="2" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995962 4431 flags.go:64] FLAG: --version="false" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995966 4431 flags.go:64] FLAG: --vmodule="" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995970 4431 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.995973 4431 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996098 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996101 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996104 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996108 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996111 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996114 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996117 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996119 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:15.005815 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996122 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996125 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996127 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996130 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996133 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996135 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996138 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996140 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996143 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996145 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996148 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996151 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996153 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996156 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996158 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996161 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996163 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996167 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996171 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:15.006383 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996174 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996178 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996182 4431 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996185 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996187 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996190 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996192 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996195 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996198 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996200 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996203 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996205 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996208 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996211 4431 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996213 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996216 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996218 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996221 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996223 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:15.006886 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996226 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996228 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996231 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996233 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996236 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996239 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996243 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996245 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996248 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996251 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996253 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996256 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996260 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996263 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996266 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996268 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996271 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996273 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996276 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:15.007376 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996279 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996281 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996284 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996286 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996289 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996292 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996294 4431 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996298 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996301 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996304 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996307 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996310 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996312 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996315 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996318 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996320 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996323 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996325 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996328 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996332 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:15.007902 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996335 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996338 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996341 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996343 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996346 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996350 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996353 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996356 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996359 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996362 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996364 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996367 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996369 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996372 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996374 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996377 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996380 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996382 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996385 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996387 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:15.008400 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996390 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:14.996414 4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996449 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996453 4431 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996456 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996459 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996462 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996464 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996467 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996469 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996472 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996474 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996478 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996481 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996483 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996486 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:15.009171 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996488 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996492 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996495 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996498 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996500 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996503 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996505 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996508 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996511 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996513 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996516 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996519 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996521 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996524 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996527 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996529 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996532 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996535 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996537 4431 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:15.009801 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996541 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996544 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996547 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996549 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996552 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996554 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996557 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996559 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996562 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996566 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996568 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996571 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996573 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996576 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996580 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996582 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996585 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996588 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996591 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996593 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:15.010276 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996596 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996599 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996601 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996604 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996607 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996609 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996612 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996615 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996617 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996620 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996622 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996625 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996627 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996630 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996632 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996635 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996638 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996640 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996643 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996646 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:15.010770 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996649 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996653 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996656 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996659 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996661 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996664 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996668 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996671 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996673 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996676 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996679 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996682 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996684 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996687 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996689 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996692 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996694 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996697 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996700 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:15.011281 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996702 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996705 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996708 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996712 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996716 4431 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996719 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996723 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996726 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996729 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996731 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996734 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996738 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996740 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:14.996743 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.001710 4431 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.001731 4431 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.001753 4431 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.001758 4431 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001832 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:15.011792 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001838 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001843 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001846 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001849 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001852 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001855 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001858 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001860 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001863 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001866 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001868 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001871 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001874 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001876 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001879 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001882 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001885 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001888 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001891 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001894 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:15.012387 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001897 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001900 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001903 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001905 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001908 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001911 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001913 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001916 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001919 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001921 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001924 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001927 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001930 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001932 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001935 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001937 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001940 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001942 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001945 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:15.013096 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001948 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001950 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001953 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001955 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001958 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001960 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001963 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001966 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001969 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001972 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001976 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001980 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001984 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001987 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001990 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001993 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001995 4431 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.001998 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002001 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:15.013554 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002003 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002006 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002009 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002012 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002014 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002017 4431 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002019 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002022 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002024 4431 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002027 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002029 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002032 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002034 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002037 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002040 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002042 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002045 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002047 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002050 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:15.014032 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002052 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002055 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002059 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002061 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002064 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002067 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002069 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002072 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002075 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002077 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002080 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002082 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002085 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002088 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002091 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002094 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002097 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002099 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002102 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002104 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:15.014542 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002107 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002109 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002112 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002115 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002117 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002120 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002122 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002125 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.002152 4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002301 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002306 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002309 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002312 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002315 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002319 4431 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002322 4431 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:15.015464 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002325 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002328 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002331 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002333 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002336 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002339 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002342 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002344 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002347 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002349 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002352 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002355 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002357 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002360 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002362 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002365 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002367 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002370 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002372 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002375 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:15.016059 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002377 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002380 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002382 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002385 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002387 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002390 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002392 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002395 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002398 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002401 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002406 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002410 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002412 4431 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002415 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002418 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002421 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002423 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002426 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002429 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:15.016586 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002431 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002434 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002436 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002439 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002441 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002444 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002446 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002449 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002452 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002454 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002457 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002459 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002462 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002465 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002467 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002469 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002472 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002474 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002477 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002479 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:15.017064 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002482 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002485 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002487 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002490 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002492 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002496 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002498 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002501 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002504 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002506 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002509 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002511 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002514 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002516 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002519 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002521 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002524 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002526 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002529 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:15.017562 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002531 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002534 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002536 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002539 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002541 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002544 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002546 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002548 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002551 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002554 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002556 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002559 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002561 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002564 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002567 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002569 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002573 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002577 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002581 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:15.018035 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002584 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002587 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.002609 4431 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002640 4431 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002644 4431 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002647 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002650 4431 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002653 4431 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002656 4431 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002658 4431 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002661 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002663 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002666 4431 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002668 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002671 4431 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002675 4431 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 13:57:15.018498 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002678 4431 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002681 4431 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002684 4431 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002687 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002689 4431 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002692 4431 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002694 4431 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002697 4431 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002699 4431 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002702 4431 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002705 4431 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002707 4431 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002711 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002713 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002716 4431 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002718 4431 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002721 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002724 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002727 4431 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 13:57:15.018899 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002729 4431 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002732 4431 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002734 4431 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002737 4431 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002739 4431 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002742 4431 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002744 4431 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002747 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002749 4431 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002752 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002754 4431 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002757 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002759 4431 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002762 4431 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002764 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002767 4431 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002769 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002771 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002789 4431 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002793 4431 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 13:57:15.019384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002797 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002799 4431 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002802 4431 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002805 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002807 4431 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002810 4431 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002813 4431 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002815 4431 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002818 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002820 4431 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002823 4431 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002826 4431 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002831 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002834 4431 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002836 4431 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002839 4431 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002841 4431 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002844 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002847 4431 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 13:57:15.020142 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002849 4431 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002852 4431 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002855 4431 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002857 4431 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002860 4431 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002862 4431 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002866 4431 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002870 4431 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002874 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002877 4431 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002879 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002882 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002885 4431 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002887 4431 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002890 4431 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002892 4431 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002895 4431 feature_gate.go:386] unrecognized feature gate: Example Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002897 4431 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002900 4431 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 13:57:15.020656 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002902 4431 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002905 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002907 4431 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002910 4431 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002914 4431 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002916 4431 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002923 4431 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002926 4431 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002929 4431 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002931 4431 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002934 4431 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002936 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002939 4431 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002941 4431 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002944 4431 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:15.002946 4431 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.002999 4431 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.003943 4431 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.003994 4431 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 13:57:15.021134 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.004169 4431 server.go:1014] "Starting client certificate rotation" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.004302 4431 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.004344 4431 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:51:35 +0000 UTC" deadline="2028-08-07 20:10:42.569530524 +0000 UTC" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.004370 4431 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15462h13m27.565162475s" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.004572 4431 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.005547 4431 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.007114 4431 log.go:25] "Validated CRI v1 runtime API" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.008629 4431 log.go:25] "Validated CRI v1 image API" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.009316 4431 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 13:57:15.021585 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.010885 4431 fs.go:135] Filesystem UUIDs: map[205ba9d1-56f0-45df-8ec7-55033244e160:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fea383b2-9528-482d-adfc-98b01e13307e:/dev/nvme0n1p4] Nov 02 13:57:15.021830 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.010906 4431 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2eb0038f417faf0433762faaeee6c659c4cec021320bfe707db383d549a0208c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2eb0038f417faf0433762faaeee6c659c4cec021320bfe707db383d549a0208c/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35e27cab57070c454541a79e8e62089fb6cbd29c3fffc9bab91083f608a872e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35e27cab57070c454541a79e8e62089fb6cbd29c3fffc9bab91083f608a872e8/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/528320711e0f4a7b9bca8b3e0a667450112dc2cc628a3a3f01da08edc94c3686/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/528320711e0f4a7b9bca8b3e0a667450112dc2cc628a3a3f01da08edc94c3686/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bb2e97ba6e566df65f539d64efbae58980c43ff99dabe163fe6d6b7e0c34b66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bb2e97ba6e566df65f539d64efbae58980c43ff99dabe163fe6d6b7e0c34b66/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d17e915d5bb8a4853e0c98aa85a36c9b32e677facd1745e8e8417a902586bd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d17e915d5bb8a4853e0c98aa85a36c9b32e677facd1745e8e8417a902586bd3/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83fd3630a95e455a63705939ac6f3566d3a1f80dca4913c6742fe8604e6519eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83fd3630a95e455a63705939ac6f3566d3a1f80dca4913c6742fe8604e6519eb/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/870177ed3be98157021524efd04da0aa1fd4bdedfd6e5f652f23634f2774d46d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/870177ed3be98157021524efd04da0aa1fd4bdedfd6e5f652f23634f2774d46d/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89e09af58061a174f016db6ce36d2b96d1bb32a5be79ee2916d0782ed61bb5f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89e09af58061a174f016db6ce36d2b96d1bb32a5be79ee2916d0782ed61bb5f0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c8436bc89fdc848a39d3b2c858f0a1f53f0a76650cfd5f9bcc55cbbe9387487/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c8436bc89fdc848a39d3b2c858f0a1f53f0a76650cfd5f9bcc55cbbe9387487/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f95bbd6e57e271c74bd542a7df7a8b721dcc5d8c584f895bdd2d0df2efd6b48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f95bbd6e57e271c74bd542a7df7a8b721dcc5d8c584f895bdd2d0df2efd6b48/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ff46100aec67fb942dd93ab86434fa0f4be9685f68d8bb3e73b6d800e4d65b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ff46100aec67fb942dd93ab86434fa0f4be9685f68d8bb3e73b6d800e4d65b3/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a0ab90615e9c97e654a386737c0e438a79f5e4d6c7ae6dfd4f6a8b783452ad4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a0ab90615e9c97e654a386737c0e438a79f5e4d6c7ae6dfd4f6a8b783452ad4/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9cb6fb493b45344844ccb77d4c8d68b49dd084ca07519d052513e7531d4cc833/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9cb6fb493b45344844ccb77d4c8d68b49dd084ca07519d052513e7531d4cc833/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a917ec2b852414f3d63b43e512bf05f1525b306efe2da1c44400180c3e43c2c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a917ec2b852414f3d63b43e512bf05f1525b306efe2da1c44400180c3e43c2c4/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/166c406a-eb7f-41ce-9d49-71f3fda95c35/volumes/kubernetes.io~projected/kube-api-access-nth9n:{mountpoint:/var/lib/kubelet/pods/166c406a-eb7f-41ce-9d49-71f3fda95c35/volumes/kubernetes.io~projected/kube-api-access-nth9n major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73/volumes/kubernetes.io~projected/kube-api-access-rshc4:{mountpoint:/var/lib/kubelet/pods/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73/volumes/kubernetes.io~projected/kube-api-access-rshc4 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2063b915-4bd7-424a-a9d1-8d8065c79166/volumes/kubernetes.io~projected/kube-api-access-lbp5h:{mountpoint:/var/lib/kubelet/pods/2063b915-4bd7-424a-a9d1-8d8065c79166/volumes/kubernetes.io~projected/kube-api-access-lbp5h major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60f37ff5-b4ee-46b6-a28c-7db9091eef76/volumes/kubernetes.io~projected/kube-api-access-bscx7:{mountpoint:/var/lib/kubelet/pods/60f37ff5-b4ee-46b6-a28c-7db9091eef76/volumes/kubernetes.io~projected/kube-api-access-bscx7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5/volumes/kubernetes.io~projected/kube-api-access-kbphq:{mountpoint:/var/lib/kubelet/pods/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5/volumes/kubernetes.io~projected/kube-api-access-kbphq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~projected/kube-api-access-6xk87:{mountpoint:/var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~projected/kube-api-access-6xk87 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0/volumes/kubernetes.io~projected/kube-api-access-hfc9n:{mountpoint:/var/lib/kubelet/pods/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0/volumes/kubernetes.io~projected/kube-api-access-hfc9n major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3206ad3-d2a7-4eec-81cb-db3a09dee305/volumes/kubernetes.io~projected/kube-api-access-fsrs9:{mountpoint:/var/lib/kubelet/pods/a3206ad3-d2a7-4eec-81cb-db3a09dee305/volumes/kubernetes.io~projected/kube-api-access-fsrs9 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9d5b255-ab45-4fce-9226-a02f7ffec197/volumes/kubernetes.io~projected/kube-api-access-tgv6d:{mountpoint:/var/lib/kubelet/pods/b9d5b255-ab45-4fce-9226-a02f7ffec197/volumes/kubernetes.io~projected/kube-api-access-tgv6d major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9d5b255-ab45-4fce-9226-a02f7ffec197/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b9d5b255-ab45-4fce-9226-a02f7ffec197/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5/volumes/kubernetes.io~projected/kube-api-access-g724w:{mountpoint:/var/lib/kubelet/pods/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5/volumes/kubernetes.io~projected/kube-api-access-g724w major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3607406-fa52-4ff4-a4e3-9a41a050d66e/volumes/kubernetes.io~projected/kube-api-access-6znb5:{mountpoint:/var/lib/kubelet/pods/d3607406-fa52-4ff4-a4e3-9a41a050d66e/volumes/kubernetes.io~projected/kube-api-access-6znb5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc38ff14-bc2b-49aa-b37e-a32248814b35/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/dc38ff14-bc2b-49aa-b37e-a32248814b35/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef68447e-3ad3-4dd2-b353-4aa00584d583/volumes/kubernetes.io~projected/kube-api-access-bxz5z:{mountpoint:/var/lib/kubelet/pods/ef68447e-3ad3-4dd2-b353-4aa00584d583/volumes/kubernetes.io~projected/kube-api-access-bxz5z major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9a86134-c854-474c-8a22-8d9732ee781a/volumes/kubernetes.io~projected/kube-api-access-c5gwq:{mountpoint:/var/lib/kubelet/pods/f9a86134-c854-474c-8a22-8d9732ee781a/volumes/kubernetes.io~projected/kube-api-access-c5gwq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volumes/kubernetes.io~projected/kube-api-access-rxw25:{mountpoint:/var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volumes/kubernetes.io~projected/kube-api-access-rxw25 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6475a1e4744069ffa633e927b139c0bf817385c8e8456586d1b74f6157a1a429/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c23eb0f7a29d54e789ee30f80de63307ff379239555383f33312b6fe958ff4b1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cd9475f8a2f455287d02aa8078e1dda29de53f66f1111d49f0f34779044590c9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/eb3ff323dd8a87e4e44d5c5838560597c7a2a0fa080afa50347b665ce47173ef/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7b44426f9b5039c0e5257f6772b6877bb1459dae91474073a4644a0b127f7865/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/aa578183a73d2b733a88a87761a44116592138708dfb326a57cf050ea43532db/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0d8f0d2e2b57f48234176e0e60a2a299a48fd1e409e8797c44da31a7a90dc839/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b7173bca7200713e615d340a5ec26304305f30194ca9cd28ed618e75c70c9046/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b29ff07738f07e5446e4e0397eac650f5c068c53599695d6bda6de15feb9ad71/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ea7cf914f11b30a09935513b126b96145c1d6bed2682914401a6fe90dc726c0a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/ee66f9aad70a579c7e4a497b06e0dc77a6d91758884a3232a294e40597b56012/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/ab1ecc1a3fcdacdc91751939c3e8d3dc78269f88c0a8dc8cdeda190ec840db40/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/d7bffd27f14a5adacc8e2026e0243af70b1c9c36b98f972f8692c79c7febc0a6/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fe5008eda7bee0bbf935f3739d8b97cc9931dc3a119d6822078b64570682275e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1f33b08ba86df8a3cff618098bb8adf79f6c155ad10427b30736a09426fb6cd0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/89241d9c81031fb2b8de0dcd8efb5e03e617f1a6841abf6c9ebd347f676a074b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2e05b4b370cd5902f76d4836f5091515f0f07122092028fa2b7adb3bab25188b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/daf747f5da05a0bafdf39578cdc57c57ecf5d70e57eca206344bdde8c0b2f614/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1d7e09fd82b3c565f99564968c5984bdb1b6cbac64c975c5d024dc34c7e95484/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a8cc2ab96300457d19fa6a776657edf23f451532f71a8851a646bbc634d89f5c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/48be3931e95fd198bd3c623d94621ee32dca8556e804c0a77049956b950e9553/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/a0246c1c6640c46594482c703777332c9be6c6ad28ab41aa0258a5adff3f7f19/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/7de301b5e5fa20a94a54a5ba842f19c9bc214aac6fcdbe93e6c84f3f9f0de533/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/566617f6f39f4b0d4c529a34892be48b91349dee22a8b4bc09a0195adc73eb87/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/86c107f8e4f8ee7fff7ad4a601713b1a304d77eaa3f289b54fe827b5dbeff76e/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/8e6338a27067ee8d2b0c69b38537b46dfe6b06edfb2066291925ae7179102f4b/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/443dddba275df36d810e68409738c7ac260732d079c847c1d08a3e4ffdb2be3e/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d523dd83519953d376af3b969e4631774a7d675582ac9f0d4ed19307b28022d4/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/a1ca2c2dfddea817dea523b2bc7724ab2a23a817cb9f40b950657fc0e800bdc7/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/a9e3aa50fcbb0868823552829f6e0e8e8c7056534321589fe7a0d1cdf2002250/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/395d0cd5d1642305b1a6b66ecaeb0c7d3267b211cd3353c12171905c241305bd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/80f3d5363eeabf61036518632752c55b3d5b13bb4ea6395e196586ffa4e6a87f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/592a1572eb5f2e8c3eddbd2673e55373326441deb63aad7c9a7b4bd24ee9e2bf/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/844116359af4e5f72773b57e433a08f510a71e56eeb779606e9c6f5753b041ca/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/326bb7e4df86d8c44ea87aeea2e79f6af5f047563f370e85d439c6a74f03f31f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/11c3baf42c72acdda20ba9408f34e7354d2c1d58bda1d3384998a2e52f2eb89d/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8d4a6d0fd045b03aded7d2f94924e8a555224ab6466e32f5006050fdb7a8badc/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4d44065922845b765e1d4d12edb0a3f8cadd3b360aa9a524d13bf819c49be741/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bb5d3799b8dbc7aca76842b8b754fe716376c16f606815cd571a184f7576f118/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018180 4431 manager.go:217] Machine: {Timestamp:2026-11-02 13:57:15.017721743 +0000 UTC m=+0.054826666 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113057 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25677aeede08c6102c79586874c11a SystemUUID:ec25677a-eede-08c6-102c-79586874c11a BootID:e9b10335-2494-48cc-9064-18e37c02197c Filesystems:[{Device:/var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f95bbd6e57e271c74bd542a7df7a8b721dcc5d8c584f895bdd2d0df2efd6b48/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9a86134-c854-474c-8a22-8d9732ee781a/volumes/kubernetes.io~projected/kube-api-access-c5gwq DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volumes/kubernetes.io~projected/kube-api-access-rxw25 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83fd3630a95e455a63705939ac6f3566d3a1f80dca4913c6742fe8604e6519eb/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~projected/kube-api-access-6xk87 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c8436bc89fdc848a39d3b2c858f0a1f53f0a76650cfd5f9bcc55cbbe9387487/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73/volumes/kubernetes.io~projected/kube-api-access-rshc4 DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/528320711e0f4a7b9bca8b3e0a667450112dc2cc628a3a3f01da08edc94c3686/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d17e915d5bb8a4853e0c98aa85a36c9b32e677facd1745e8e8417a902586bd3/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60f37ff5-b4ee-46b6-a28c-7db9091eef76/volumes/kubernetes.io~projected/kube-api-access-bscx7 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2063b915-4bd7-424a-a9d1-8d8065c79166/volumes/kubernetes.io~projected/kube-api-access-lbp5h DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a917ec2b852414f3d63b43e512bf05f1525b306efe2da1c44400180c3e43c2c4/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0/volumes/kubernetes.io~projected/kube-api-access-hfc9n DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2eb0038f417faf0433762faaeee6c659c4cec021320bfe707db383d549a0208c/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89e09af58061a174f016db6ce36d2b96d1bb32a5be79ee2916d0782ed61bb5f0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/870177ed3be98157021524efd04da0aa1fd4bdedfd6e5f652f23634f2774d46d/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3206ad3-d2a7-4eec-81cb-db3a09dee305/volumes/kubernetes.io~projected/kube-api-access-fsrs9 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef68447e-3ad3-4dd2-b353-4aa00584d583/volumes/kubernetes.io~projected/kube-api-access-bxz5z DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3607406-fa52-4ff4-a4e3-9a41a050d66e/volumes/kubernetes.io~projected/kube-api-access-6znb5 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9d5b255-ab45-4fce-9226-a02f7ffec197/volumes/kubernetes.io~projected/kube-api-access-tgv6d DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5/volumes/kubernetes.io~projected/kube-api-access-g724w DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35e27cab57070c454541a79e8e62089fb6cbd29c3fffc9bab91083f608a872e8/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/75eff6b2-9975-4809-a676-f78efa1d45fa/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bb2e97ba6e566df65f539d64efbae58980c43ff99dabe163fe6d6b7e0c34b66/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9d5b255-ab45-4fce-9226-a02f7ffec197/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a0ab90615e9c97e654a386737c0e438a79f5e4d6c7ae6dfd4f6a8b783452ad4/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fdee392a-98f2-4018-ba57-23a7bc2952b7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc38ff14-bc2b-49aa-b37e-a32248814b35/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ff46100aec67fb942dd93ab86434fa0f4be9685f68d8bb3e73b6d800e4d65b3/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/166c406a-eb7f-41ce-9d49-71f3fda95c35/volumes/kubernetes.io~projected/kube-api-access-nth9n DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5/volumes/kubernetes.io~projected/kube-api-access-kbphq DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9cb6fb493b45344844ccb77d4c8d68b49dd084ca07519d052513e7531d4cc833/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:528320711e0f4a7 MacAddress:72:b0:ce:55:60:65 Speed:10000 Mtu:8901} {Name:8ff46100aec67fb MacAddress:52:46:70:cb:5a:2a Speed:10000 Mtu:8901} {Name:9a0ab90615e9c97 MacAddress:16:d7:0d:29:86:e9 Speed:10000 Mtu:8901} {Name:9cb6fb493b45344 MacAddress:66:9d:0b:e1:19:a8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e9:4d:c6:b1 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:1a:9d:4e:41:00 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:e9:4d:c6:b1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:2a:87:92:13:fe:c2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:2d:19:89:4c:bc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018417 4431 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018456 4431 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018695 4431 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018715 4431 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018842 4431 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018850 4431 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018863 4431 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018875 4431 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.018944 4431 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019090 4431 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019183 4431 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019212 4431 kubelet.go:493] "Attempting to sync node with API server" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019221 4431 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019230 4431 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019238 4431 kubelet.go:405] "Adding apiserver pod source" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019245 4431 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019787 4431 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.019796 4431 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:57:15.021968 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.020271 4431 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 13:57:15.022557 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022242 4431 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 13:57:15.022557 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022251 4431 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 13:57:15.022557 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022342 4431 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 13:57:15.022557 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022522 4431 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 13:57:15.022557 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022551 4431 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 13:57:15.022690 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022650 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 13:57:15.022690 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022664 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 13:57:15.022690 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022673 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 13:57:15.022690 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022682 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 13:57:15.022690 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022690 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022698 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022703 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022712 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022726 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022733 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022749 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022759 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022797 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 13:57:15.022893 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.022803 4431 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 13:57:15.025839 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.025804 4431 server.go:1263] "Started kubelet" Nov 02 13:57:15.025918 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.025888 4431 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 13:57:15.026034 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.025942 4431 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 13:57:15.026079 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.026065 4431 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 13:57:15.026315 ip-10-0-134-9 systemd[1]: Started Kubernetes Kubelet. Nov 02 13:57:15.026629 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.026479 4431 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 13:57:15.027500 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.027466 4431 server.go:317] "Adding debug handlers to kubelet server" Nov 02 13:57:15.029496 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.029476 4431 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 13:57:15.030211 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.030119 4431 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 13:57:15.030411 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.030344 4431 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:51:37 +0000 UTC" deadline="2028-05-01 08:07:27.364963351 +0000 UTC" Nov 02 13:57:15.030411 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.030385 4431 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13098h10m12.334581441s" Nov 02 13:57:15.031253 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:15.030437 4431 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Nov 02 13:57:15.031253 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.030484 4431 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 13:57:15.031253 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.030493 4431 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 13:57:15.031253 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.030723 4431 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032270 4431 factory.go:55] Registering systemd factory Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032288 4431 factory.go:223] Registration of the systemd container factory successfully Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032602 4431 factory.go:153] Registering CRI-O factory Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032614 4431 factory.go:223] Registration of the crio container factory successfully Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032676 4431 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032698 4431 factory.go:103] Registering Raw factory Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032711 4431 manager.go:1199] Started watching for new ooms in manager Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032860 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2063b915-4bd7-424a-a9d1-8d8065c79166" volumeName="kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-cni-binary-copy" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032919 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6efd880e-a7eb-4c81-b081-f3a2ca09e1c5" volumeName="kubernetes.io/projected/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5-kube-api-access-kbphq" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032936 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ba6035a-7b2a-46ab-9506-8fd9b3d369a0" volumeName="kubernetes.io/projected/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-kube-api-access-hfc9n" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032951 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d5b255-ab45-4fce-9226-a02f7ffec197" volumeName="kubernetes.io/secret/b9d5b255-ab45-4fce-9226-a02f7ffec197-cert" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032968 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef68447e-3ad3-4dd2-b353-4aa00584d583" volumeName="kubernetes.io/empty-dir/ef68447e-3ad3-4dd2-b353-4aa00584d583-tmp-dir" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032981 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdee392a-98f2-4018-ba57-23a7bc2952b7" volumeName="kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-config" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.032997 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60f37ff5-b4ee-46b6-a28c-7db9091eef76" volumeName="kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033010 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a1d83b-52ea-4fcd-9843-4eb8199d3d65" volumeName="kubernetes.io/configmap/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-konnectivity-ca" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033027 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2063b915-4bd7-424a-a9d1-8d8065c79166" volumeName="kubernetes.io/projected/2063b915-4bd7-424a-a9d1-8d8065c79166-kube-api-access-lbp5h" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033048 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60f37ff5-b4ee-46b6-a28c-7db9091eef76" volumeName="kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033064 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3607406-fa52-4ff4-a4e3-9a41a050d66e" volumeName="kubernetes.io/configmap/d3607406-fa52-4ff4-a4e3-9a41a050d66e-iptables-alerter-script" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033082 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdee392a-98f2-4018-ba57-23a7bc2952b7" volumeName="kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-env-overrides" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033097 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60f37ff5-b4ee-46b6-a28c-7db9091eef76" volumeName="kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-binary-copy" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033117 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75eff6b2-9975-4809-a676-f78efa1d45fa" volumeName="kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-tmp" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033151 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75eff6b2-9975-4809-a676-f78efa1d45fa" volumeName="kubernetes.io/projected/75eff6b2-9975-4809-a676-f78efa1d45fa-kube-api-access-6xk87" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033167 4431 manager.go:319] Starting recovery of all containers Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033173 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdee392a-98f2-4018-ba57-23a7bc2952b7" volumeName="kubernetes.io/projected/fdee392a-98f2-4018-ba57-23a7bc2952b7-kube-api-access-rxw25" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033216 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef68447e-3ad3-4dd2-b353-4aa00584d583" volumeName="kubernetes.io/configmap/ef68447e-3ad3-4dd2-b353-4aa00584d583-config-volume" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033233 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73" volumeName="kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033251 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2063b915-4bd7-424a-a9d1-8d8065c79166" volumeName="kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-daemon-config" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033265 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3206ad3-d2a7-4eec-81cb-db3a09dee305" volumeName="kubernetes.io/projected/a3206ad3-d2a7-4eec-81cb-db3a09dee305-kube-api-access-fsrs9" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033289 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5" volumeName="kubernetes.io/projected/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-kube-api-access-g724w" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033302 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3607406-fa52-4ff4-a4e3-9a41a050d66e" volumeName="kubernetes.io/projected/d3607406-fa52-4ff4-a4e3-9a41a050d66e-kube-api-access-6znb5" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033316 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc38ff14-bc2b-49aa-b37e-a32248814b35" volumeName="kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033336 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3a1d83b-52ea-4fcd-9843-4eb8199d3d65" volumeName="kubernetes.io/secret/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-agent-certs" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033351 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60f37ff5-b4ee-46b6-a28c-7db9091eef76" volumeName="kubernetes.io/projected/60f37ff5-b4ee-46b6-a28c-7db9091eef76-kube-api-access-bscx7" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033370 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75eff6b2-9975-4809-a676-f78efa1d45fa" volumeName="kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-tuned" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033382 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9d5b255-ab45-4fce-9226-a02f7ffec197" volumeName="kubernetes.io/projected/b9d5b255-ab45-4fce-9226-a02f7ffec197-kube-api-access-tgv6d" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033397 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5" volumeName="kubernetes.io/empty-dir/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-tmp-dir" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033431 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef68447e-3ad3-4dd2-b353-4aa00584d583" volumeName="kubernetes.io/projected/ef68447e-3ad3-4dd2-b353-4aa00584d583-kube-api-access-bxz5z" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033448 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdee392a-98f2-4018-ba57-23a7bc2952b7" volumeName="kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-script-lib" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033460 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c406a-eb7f-41ce-9d49-71f3fda95c35" volumeName="kubernetes.io/empty-dir/166c406a-eb7f-41ce-9d49-71f3fda95c35-data-volume" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033471 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="166c406a-eb7f-41ce-9d49-71f3fda95c35" volumeName="kubernetes.io/projected/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-api-access-nth9n" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033490 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3206ad3-d2a7-4eec-81cb-db3a09dee305" volumeName="kubernetes.io/configmap/a3206ad3-d2a7-4eec-81cb-db3a09dee305-serviceca" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033504 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9a86134-c854-474c-8a22-8d9732ee781a" volumeName="kubernetes.io/projected/f9a86134-c854-474c-8a22-8d9732ee781a-kube-api-access-c5gwq" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033521 4431 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdee392a-98f2-4018-ba57-23a7bc2952b7" volumeName="kubernetes.io/secret/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033533 4431 reconstruct.go:97] "Volume reconstruction finished" Nov 02 13:57:15.034225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.033540 4431 reconciler.go:29] "Reconciler: start to sync state" Nov 02 13:57:15.065336 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.065302 4431 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 13:57:15.067844 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.067820 4431 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 13:57:15.067844 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.067844 4431 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 13:57:15.068017 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.067862 4431 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 13:57:15.068017 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:15.067905 4431 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 13:57:15.075713 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.075681 4431 generic.go:356] "Generic (PLEG): container finished" podID="04b7d7036f47f423ed3f88aa11798947" containerID="ff23928bbd43372e67561800bc60ab15d29720d97ad3b623551bccfad8653a49" exitCode=0 Nov 02 13:57:15.089368 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.089342 4431 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="9c273344a6a9bfc649934977cedc20689d2b6685a680845be88a9eb20394a630" exitCode=0 Nov 02 13:57:15.089368 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.089367 4431 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="74384ba9ba4046c6f3c0c912da35d059f21b388b4613d7e634b37ba9344e62c9" exitCode=0 Nov 02 13:57:15.089484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.089381 4431 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="521454d97ea5cec9fd548d8a725f8372dc2dcc68ce347ef2cc36d47a2461cea4" exitCode=0 Nov 02 13:57:15.089484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.089391 4431 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="f2547009e6a1659c3ce8215765edf1b3156bfb1d46f990278452772b103cb0fc" exitCode=0 Nov 02 13:57:15.089484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.089401 4431 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="20a3052ad233be5429eb9a36657987a0046bef8c279478a021ec4539b5df6a61" exitCode=0 Nov 02 13:57:15.089484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.089411 4431 generic.go:356] "Generic (PLEG): container finished" podID="60f37ff5-b4ee-46b6-a28c-7db9091eef76" containerID="8b538b719ddadfe1323ee3d1218728dc206a8a27f32c05fadb3f4bfcc8ca7359" exitCode=0 Nov 02 13:57:15.092829 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.092806 4431 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 13:57:15.093727 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.093698 4431 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:57:15.104702 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.104686 4431 manager.go:324] Recovery completed Nov 02 13:57:15.107195 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.107173 4431 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:57:15.107285 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.107224 4431 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:57:15.107337 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.107320 4431 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 13:57:15.118389 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118373 4431 cpu_manager.go:226] "Starting" policy="none" Nov 02 13:57:15.118389 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118386 4431 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 13:57:15.118489 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118401 4431 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 13:57:15.118536 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118504 4431 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 13:57:15.118536 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118514 4431 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 13:57:15.118536 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118529 4431 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 13:57:15.118536 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118534 4431 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 13:57:15.118661 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118539 4431 policy_none.go:50] "Start" Nov 02 13:57:15.118661 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118547 4431 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 13:57:15.118661 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118555 4431 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 13:57:15.118750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118709 4431 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 13:57:15.118750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118716 4431 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 13:57:15.118750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.118722 4431 policy_none.go:44] "Start" Nov 02 13:57:15.122485 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.122458 4431 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 13:57:15.125230 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.125214 4431 manager.go:341] "Starting Device Plugin manager" Nov 02 13:57:15.125935 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:15.125247 4431 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 13:57:15.125994 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.125939 4431 server.go:87] "Starting device plugin registration server" Nov 02 13:57:15.126252 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.126239 4431 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 13:57:15.126282 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.126255 4431 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 13:57:15.126355 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.126334 4431 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 13:57:15.126465 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.126423 4431 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 13:57:15.126465 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.126460 4431 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 13:57:15.126465 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.126467 4431 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 13:57:15.126623 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.126500 4431 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:57:15.126449863Z","UUID":"75ee33cf-8d4a-43e4-9e8a-3d8db6ba9fac","Handler":null,"Name":"","Endpoint":""} Nov 02 13:57:15.127565 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:15.127545 4431 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 13:57:15.127667 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.127649 4431 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 13:57:15.127709 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.127678 4431 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 13:57:15.168639 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.168573 4431 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal"] Nov 02 13:57:15.169028 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.169003 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"04b7d7036f47f423ed3f88aa11798947","Type":"ContainerStarted","Data":"15726d02f4f55e280d3d7330bf6c7bda1eec99175775f1979a08832afb3f2b10"} Nov 02 13:57:15.169080 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.169035 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"04b7d7036f47f423ed3f88aa11798947","Type":"ContainerDied","Data":"ff23928bbd43372e67561800bc60ab15d29720d97ad3b623551bccfad8653a49"} Nov 02 13:57:15.169080 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.169050 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"04b7d7036f47f423ed3f88aa11798947","Type":"ContainerStarted","Data":"870177ed3be98157021524efd04da0aa1fd4bdedfd6e5f652f23634f2774d46d"} Nov 02 13:57:15.169080 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.169062 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" event={"ID":"6e4f9ec4cb1fc6480c545594116d855c","Type":"ContainerStarted","Data":"31ee5d4ca175d67b4bc46b94a91ee22b1b377175c9b9b6ee829cc77a0f7624ff"} Nov 02 13:57:15.169080 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.169073 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" event={"ID":"6e4f9ec4cb1fc6480c545594116d855c","Type":"ContainerStarted","Data":"7d17e915d5bb8a4853e0c98aa85a36c9b32e677facd1745e8e8417a902586bd3"} Nov 02 13:57:15.169192 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.169176 4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.169263 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.169176 4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.193739 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.193717 4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:57:15.193821 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:15.193760 4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.197188 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.197174 4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:57:15.197234 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:15.197211 4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.226743 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.226722 4431 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 13:57:15.227836 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.227822 4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Nov 02 13:57:15.227897 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.227849 4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Nov 02 13:57:15.227897 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.227859 4431 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Nov 02 13:57:15.227897 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.227885 4431 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.270972 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.270949 4431 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.271044 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.270995 4431 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.294654 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.294631 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.294736 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.294657 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.294736 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.294676 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4f9ec4cb1fc6480c545594116d855c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"6e4f9ec4cb1fc6480c545594116d855c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.395280 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.395257 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.395349 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.395314 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.395394 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.395349 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.395394 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.395368 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b7d7036f47f423ed3f88aa11798947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"04b7d7036f47f423ed3f88aa11798947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.395453 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.395400 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4f9ec4cb1fc6480c545594116d855c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"6e4f9ec4cb1fc6480c545594116d855c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:57:15.395453 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:15.395430 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e4f9ec4cb1fc6480c545594116d855c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"6e4f9ec4cb1fc6480c545594116d855c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:57:16.019391 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.019354 4431 apiserver.go:52] "Watching apiserver" Nov 02 13:57:16.062982 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.062958 4431 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 13:57:16.063542 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.063524 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm","openshift-ovn-kubernetes/ovnkube-node-zx5gw","openshift-dns/node-resolver-kjcwb","openshift-network-operator/iptables-alerter-r7fcg","kube-system/global-pull-secret-syncer-ndjmx","openshift-cluster-node-tuning-operator/tuned-pv4lc","openshift-network-diagnostics/network-check-target-tqb8d","openshift-multus/multus-additional-cni-plugins-wpzw4","kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal","kube-system/konnectivity-agent-5q6hs","openshift-ingress-canary/ingress-canary-4q2dh","openshift-multus/multus-w8qtw","openshift-insights/insights-runtime-extractor-vwqt2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72","openshift-image-registry/node-ca-dtc82","openshift-multus/network-metrics-daemon-bjbp9","openshift-dns/dns-default-bv8r5"] Nov 02 13:57:16.069907 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.069878 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:16.071285 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.071266 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.071894 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.071877 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.082936 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.082914 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.083046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.083033 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 13:57:16.083097 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.083049 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-cd6m4\"" Nov 02 13:57:16.083097 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.083064 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 13:57:16.083168 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.083153 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 13:57:16.083204 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.083176 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 13:57:16.095218 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.095197 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 13:57:16.095533 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.095508 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 13:57:16.095599 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.095531 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 13:57:16.095599 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.095538 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 13:57:16.097212 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097195 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 13:57:16.097306 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097287 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mgxc\"" Nov 02 13:57:16.097458 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097439 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.097458 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097449 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.097627 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097490 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-75928\"" Nov 02 13:57:16.097627 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097596 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.097737 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097719 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 13:57:16.097810 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097729 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.097917 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097901 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.097987 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097933 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.097987 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097944 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.097987 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097960 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.097987 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097972 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kftg9\"" Nov 02 13:57:16.097987 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097973 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 13:57:16.098206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.097996 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8ms52\"" Nov 02 13:57:16.098206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098037 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-sczcr\"" Nov 02 13:57:16.098206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098119 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 13:57:16.098206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098133 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.098206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098161 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.098206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098194 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 13:57:16.098206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098204 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098291 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4wpm\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098344 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098457 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098492 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gjwh\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098525 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098529 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlrs4\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098550 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098559 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xm2nq\"" Nov 02 13:57:16.098563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098566 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m5vzv\"" Nov 02 13:57:16.098905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098559 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.098905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098555 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.098905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098610 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 13:57:16.098905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098717 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9rw8\"" Nov 02 13:57:16.098905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098816 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.098905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098845 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5mnrf\"" Nov 02 13:57:16.098905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098896 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 13:57:16.099128 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098927 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 13:57:16.099128 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098933 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.099128 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098942 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.099128 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.098959 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.099128 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.099027 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 13:57:16.099261 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.099175 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 13:57:16.099261 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.099199 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2zrsm\"" Nov 02 13:57:16.099261 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.099217 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 13:57:16.102145 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.102131 4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:57:16.102245 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.102234 4431 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:16.109188 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.109172 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xjjmq\"" Nov 02 13:57:16.109249 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.109223 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9cgsx\"" Nov 02 13:57:16.109249 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.109225 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 13:57:16.132636 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.132619 4431 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 13:57:16.137873 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.137844 4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:57:16.137920 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.137896 4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Nov 02 13:57:16.138843 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.138831 4431 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 13:57:16.138879 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.138860 4431 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Nov 02 13:57:16.199037 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199010 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-conf-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.199037 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199038 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-tmp\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.199209 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199054 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrs9\" (UniqueName: \"kubernetes.io/projected/a3206ad3-d2a7-4eec-81cb-db3a09dee305-kube-api-access-fsrs9\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:57:16.199209 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199070 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-kubelet\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.199209 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199153 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cnibin\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.199209 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199200 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-tmp\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.199405 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199271 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-hostroot\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.199458 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199403 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-sys\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.199458 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199430 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-systemd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.199458 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199453 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-ovn\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.199626 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199495 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-log-socket\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.199626 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199533 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-script-lib\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.199626 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199581 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:16.199766 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199647 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-modprobe-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.199766 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199688 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-config\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.199766 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199734 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.199880 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199841 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-script-lib\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.199880 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199868 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-socket-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.199981 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199915 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9d5b255-ab45-4fce-9226-a02f7ffec197-cert\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:16.199981 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199941 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.200076 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.199984 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-netns\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.200076 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200026 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g724w\" (UniqueName: \"kubernetes.io/projected/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-kube-api-access-g724w\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:57:16.200076 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200054 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-etc-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.200197 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200075 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovnkube-config\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.200197 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200174 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.200329 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200193 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9d5b255-ab45-4fce-9226-a02f7ffec197-cert\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:16.200329 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200203 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.200329 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200226 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.200329 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200246 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-cnibin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.200329 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200284 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-multus-certs\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.200329 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200300 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.200490 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200339 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-etc-kubernetes\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.200490 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200360 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-run\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.200490 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200381 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.200490 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200400 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-system-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.200490 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200418 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-bin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.200490 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200434 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbp5h\" (UniqueName: \"kubernetes.io/projected/2063b915-4bd7-424a-a9d1-8d8065c79166-kube-api-access-lbp5h\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.200490 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200477 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-var-lib-kubelet\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.200679 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200608 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-host\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.200679 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200635 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3206ad3-d2a7-4eec-81cb-db3a09dee305-host\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:57:16.200679 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200650 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-systemd-units\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.200767 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200678 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-sys-fs\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.200767 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200720 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-bin\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.200767 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200747 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-env-overrides\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.200889 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200773 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-binary-copy\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.200931 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200918 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdee392a-98f2-4018-ba57-23a7bc2952b7-env-overrides\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.200960 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200940 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-conf\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.200960 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200954 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cni-binary-copy\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.201018 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.200988 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-netns\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.201046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201025 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-var-lib-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.201046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201043 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.201101 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201068 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.201101 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201092 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef68447e-3ad3-4dd2-b353-4aa00584d583-tmp-dir\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.201155 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201107 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-node-log\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.201155 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201141 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-registration-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.201207 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201162 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef68447e-3ad3-4dd2-b353-4aa00584d583-tmp-dir\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.201207 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201170 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgv6d\" (UniqueName: \"kubernetes.io/projected/b9d5b255-ab45-4fce-9226-a02f7ffec197-kube-api-access-tgv6d\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:16.201207 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201186 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.201287 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201242 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-tuned\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.201315 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201286 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-tmp-dir\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:57:16.201361 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201343 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/60f37ff5-b4ee-46b6-a28c-7db9091eef76-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.201415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201356 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-kubelet-config\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:16.201415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201349 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-tuned\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.201415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201396 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbphq\" (UniqueName: \"kubernetes.io/projected/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5-kube-api-access-kbphq\") pod \"migrator-66df5d496b-4xbqm\" (UID: \"6efd880e-a7eb-4c81-b081-f3a2ca09e1c5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" Nov 02 13:57:16.201555 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201415 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-tmp-dir\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:57:16.201555 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201433 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-system-cni-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.201555 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201518 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:16.201555 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201542 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysconfig\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.201750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201567 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-kubernetes\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.201750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201585 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xk87\" (UniqueName: \"kubernetes.io/projected/75eff6b2-9975-4809-a676-f78efa1d45fa-kube-api-access-6xk87\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.201750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201619 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-slash\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.201750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201640 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-etc-selinux\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.201750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201685 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfc9n\" (UniqueName: \"kubernetes.io/projected/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-kube-api-access-hfc9n\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:16.201750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201716 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.201750 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201744 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-socket-dir-parent\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.202009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201808 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-multus\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.202009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201845 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-systemd\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.202009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201892 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-hosts-file\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:57:16.202009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201922 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3607406-fa52-4ff4-a4e3-9a41a050d66e-host-slash\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:57:16.202009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201960 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxw25\" (UniqueName: \"kubernetes.io/projected/fdee392a-98f2-4018-ba57-23a7bc2952b7-kube-api-access-rxw25\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.202009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.201980 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-konnectivity-ca\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:57:16.202009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202007 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovn-node-metrics-cert\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.202307 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202023 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.202307 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202040 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nth9n\" (UniqueName: \"kubernetes.io/projected/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-api-access-nth9n\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.202307 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202058 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68447e-3ad3-4dd2-b353-4aa00584d583-config-volume\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.202307 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202074 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-kubelet\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.202307 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202090 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d3607406-fa52-4ff4-a4e3-9a41a050d66e-iptables-alerter-script\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:57:16.202307 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202111 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:16.202307 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202143 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-netd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.202307 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202235 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-agent-certs\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202344 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef68447e-3ad3-4dd2-b353-4aa00584d583-config-volume\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202363 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc38ff14-bc2b-49aa-b37e-a32248814b35-original-pull-secret\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202388 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscx7\" (UniqueName: \"kubernetes.io/projected/60f37ff5-b4ee-46b6-a28c-7db9091eef76-kube-api-access-bscx7\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202407 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/166c406a-eb7f-41ce-9d49-71f3fda95c35-data-volume\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202441 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxz5z\" (UniqueName: \"kubernetes.io/projected/ef68447e-3ad3-4dd2-b353-4aa00584d583-kube-api-access-bxz5z\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202466 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-cni-binary-copy\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202490 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-lib-modules\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202512 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3206ad3-d2a7-4eec-81cb-db3a09dee305-serviceca\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202539 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-device-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202567 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gwq\" (UniqueName: \"kubernetes.io/projected/f9a86134-c854-474c-8a22-8d9732ee781a-kube-api-access-c5gwq\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202570 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-konnectivity-ca\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202583 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/166c406a-eb7f-41ce-9d49-71f3fda95c35-data-volume\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202593 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/166c406a-eb7f-41ce-9d49-71f3fda95c35-crio-socket\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202602 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d3607406-fa52-4ff4-a4e3-9a41a050d66e-iptables-alerter-script\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202618 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-k8s-cni-cncf-io\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.202665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202638 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3a1d83b-52ea-4fcd-9843-4eb8199d3d65-agent-certs\") pod \"konnectivity-agent-5q6hs\" (UID: \"e3a1d83b-52ea-4fcd-9843-4eb8199d3d65\") " pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:57:16.203228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202685 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-daemon-config\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.203228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202718 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.203228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202735 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-cni-binary-copy\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.203228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202818 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znb5\" (UniqueName: \"kubernetes.io/projected/d3607406-fa52-4ff4-a4e3-9a41a050d66e-kube-api-access-6znb5\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:57:16.203228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202838 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-dbus\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:16.203228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202855 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3206ad3-d2a7-4eec-81cb-db3a09dee305-serviceca\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:57:16.203228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202888 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-daemon-config\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.203228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.202922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdee392a-98f2-4018-ba57-23a7bc2952b7-ovn-node-metrics-cert\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.221725 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.221700 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrs9\" (UniqueName: \"kubernetes.io/projected/a3206ad3-d2a7-4eec-81cb-db3a09dee305-kube-api-access-fsrs9\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:57:16.222264 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.222248 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshc4\" (UniqueName: \"kubernetes.io/projected/1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73-kube-api-access-rshc4\") pod \"network-check-target-tqb8d\" (UID: \"1ba2eeed-bf6d-4c29-9505-d8a8f9d53e73\") " pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:16.223129 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.223110 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbp5h\" (UniqueName: \"kubernetes.io/projected/2063b915-4bd7-424a-a9d1-8d8065c79166-kube-api-access-lbp5h\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.223265 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.223249 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g724w\" (UniqueName: \"kubernetes.io/projected/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-kube-api-access-g724w\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:57:16.224703 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.224684 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbphq\" (UniqueName: \"kubernetes.io/projected/6efd880e-a7eb-4c81-b081-f3a2ca09e1c5-kube-api-access-kbphq\") pod \"migrator-66df5d496b-4xbqm\" (UID: \"6efd880e-a7eb-4c81-b081-f3a2ca09e1c5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4xbqm" Nov 02 13:57:16.224848 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.224790 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xk87\" (UniqueName: \"kubernetes.io/projected/75eff6b2-9975-4809-a676-f78efa1d45fa-kube-api-access-6xk87\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.225056 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.225037 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfc9n\" (UniqueName: \"kubernetes.io/projected/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-kube-api-access-hfc9n\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:16.225437 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.225420 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxw25\" (UniqueName: \"kubernetes.io/projected/fdee392a-98f2-4018-ba57-23a7bc2952b7-kube-api-access-rxw25\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.228145 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.227857 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgv6d\" (UniqueName: \"kubernetes.io/projected/b9d5b255-ab45-4fce-9226-a02f7ffec197-kube-api-access-tgv6d\") pod \"ingress-canary-4q2dh\" (UID: \"b9d5b255-ab45-4fce-9226-a02f7ffec197\") " pod="openshift-ingress-canary/ingress-canary-4q2dh" Nov 02 13:57:16.234245 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.234223 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znb5\" (UniqueName: \"kubernetes.io/projected/d3607406-fa52-4ff4-a4e3-9a41a050d66e-kube-api-access-6znb5\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:57:16.234582 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.234560 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscx7\" (UniqueName: \"kubernetes.io/projected/60f37ff5-b4ee-46b6-a28c-7db9091eef76-kube-api-access-bscx7\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.235203 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.235183 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxz5z\" (UniqueName: \"kubernetes.io/projected/ef68447e-3ad3-4dd2-b353-4aa00584d583-kube-api-access-bxz5z\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.235272 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.235227 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gwq\" (UniqueName: \"kubernetes.io/projected/f9a86134-c854-474c-8a22-8d9732ee781a-kube-api-access-c5gwq\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.238604 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.238588 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nth9n\" (UniqueName: \"kubernetes.io/projected/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-api-access-nth9n\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.303141 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303080 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-kubelet\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303141 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303119 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cnibin\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.303141 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303138 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-hostroot\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303161 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-sys\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303184 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-systemd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303187 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-kubelet\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303200 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-cnibin\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303218 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-hostroot\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303237 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-ovn\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303233 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-sys\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303207 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-ovn\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303249 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-systemd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303277 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-log-socket\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303299 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-log-socket\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303301 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-modprobe-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303324 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303341 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-socket-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303355 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-netns\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303371 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-etc-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303403 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303415 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-netns\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303416 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303427 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303439 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-socket-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303446 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303449 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303471 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-cnibin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303473 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-etc-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303481 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-modprobe-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303497 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-multus-certs\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303522 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-etc-kubernetes\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303526 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.303525 4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303525 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-multus-certs\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303546 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-run\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303560 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-etc-kubernetes\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303528 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-cnibin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303575 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-run\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303583 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.303594 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.803580259 +0000 UTC m=+1.840685170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303612 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303616 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-system-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303640 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-bin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303655 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-var-lib-kubelet\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303666 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-system-cni-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.303662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303668 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-host\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303688 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3206ad3-d2a7-4eec-81cb-db3a09dee305-host\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303670 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-bin\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303702 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-host\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303708 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-var-lib-kubelet\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303719 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-systemd-units\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303741 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-sys-fs\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303748 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3206ad3-d2a7-4eec-81cb-db3a09dee305-host\") pod \"node-ca-dtc82\" (UID: \"a3206ad3-d2a7-4eec-81cb-db3a09dee305\") " pod="openshift-image-registry/node-ca-dtc82" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303758 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-bin\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303791 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-conf\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303803 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-sys-fs\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303803 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-systemd-units\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303809 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-netns\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303829 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-run-netns\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303835 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-var-lib-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303862 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303819 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-bin\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303879 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303891 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-conf\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303896 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-node-log\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303865 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-var-lib-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.303907 4431 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303916 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-registration-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303921 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-node-log\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303963 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-registration-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.303966 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.803951782 +0000 UTC m=+1.841056710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : configmap "kube-rbac-proxy" not found Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.303976 4431 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.303992 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-kubelet-config\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.304016 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.804008779 +0000 UTC m=+1.841113691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304036 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-kubelet-config\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304028 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-system-cni-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304056 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304062 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-system-cni-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304077 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysconfig\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304092 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-kubernetes\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304107 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-slash\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304121 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-etc-selinux\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304137 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysconfig\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304144 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304168 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-socket-dir-parent\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304180 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-slash\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304183 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-kubernetes\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304192 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-multus\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.304201 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304212 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-run-openvswitch\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304234 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-systemd\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304221 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-cni-multus\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304239 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-socket-dir-parent\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.304249 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:16.804237623 +0000 UTC m=+1.841342534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : secret "metrics-daemon-secret" not found Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304194 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-etc-selinux\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304282 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-systemd\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304289 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-hosts-file\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304306 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3607406-fa52-4ff4-a4e3-9a41a050d66e-host-slash\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304323 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304339 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-kubelet\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304359 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-var-lib-kubelet\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304358 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-netd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304382 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdee392a-98f2-4018-ba57-23a7bc2952b7-host-cni-netd\") pod \"ovnkube-node-zx5gw\" (UID: \"fdee392a-98f2-4018-ba57-23a7bc2952b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304379 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3607406-fa52-4ff4-a4e3-9a41a050d66e-host-slash\") pod \"iptables-alerter-r7fcg\" (UID: \"d3607406-fa52-4ff4-a4e3-9a41a050d66e\") " pod="openshift-network-operator/iptables-alerter-r7fcg" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304368 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5-hosts-file\") pod \"node-resolver-kjcwb\" (UID: \"c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5\") " pod="openshift-dns/node-resolver-kjcwb" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304398 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60f37ff5-b4ee-46b6-a28c-7db9091eef76-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpzw4\" (UID: \"60f37ff5-b4ee-46b6-a28c-7db9091eef76\") " pod="openshift-multus/multus-additional-cni-plugins-wpzw4" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304401 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-lib-modules\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304430 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-device-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304446 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/166c406a-eb7f-41ce-9d49-71f3fda95c35-crio-socket\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304459 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-k8s-cni-cncf-io\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304471 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-lib-modules\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304481 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304492 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9a86134-c854-474c-8a22-8d9732ee781a-device-dir\") pod \"aws-ebs-csi-driver-node-47z72\" (UID: \"f9a86134-c854-474c-8a22-8d9732ee781a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-47z72" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304505 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-dbus\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304504 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/166c406a-eb7f-41ce-9d49-71f3fda95c35-crio-socket\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304498 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-host-run-k8s-cni-cncf-io\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304527 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-conf-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304560 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75eff6b2-9975-4809-a676-f78efa1d45fa-etc-sysctl-d\") pod \"tuned-pv4lc\" (UID: \"75eff6b2-9975-4809-a676-f78efa1d45fa\") " pod="openshift-cluster-node-tuning-operator/tuned-pv4lc" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304569 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc38ff14-bc2b-49aa-b37e-a32248814b35-dbus\") pod \"global-pull-secret-syncer-ndjmx\" (UID: \"dc38ff14-bc2b-49aa-b37e-a32248814b35\") " pod="kube-system/global-pull-secret-syncer-ndjmx" Nov 02 13:57:16.304673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.304590 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2063b915-4bd7-424a-a9d1-8d8065c79166-multus-conf-dir\") pod \"multus-w8qtw\" (UID: \"2063b915-4bd7-424a-a9d1-8d8065c79166\") " pod="openshift-multus/multus-w8qtw" Nov 02 13:57:16.370503 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.370479 4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5q6hs_e3a1d83b-52ea-4fcd-9843-4eb8199d3d65/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:16.520188 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.520161 4431 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5q6hs_e3a1d83b-52ea-4fcd-9843-4eb8199d3d65/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:16.808828 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.808797 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.809002 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.808838 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:16.809002 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.808859 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:16.809002 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.808904 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:16.809002 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.808980 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:16.809202 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.809048 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.80903596 +0000 UTC m=+2.846140898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : secret "metrics-daemon-secret" not found Nov 02 13:57:16.809202 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.808982 4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:16.809202 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.809125 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.809112074 +0000 UTC m=+2.846216984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:16.809202 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.808982 4431 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:16.809202 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:16.809162 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:17.809152258 +0000 UTC m=+2.846257168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:16.809443 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:16.809346 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/166c406a-eb7f-41ce-9d49-71f3fda95c35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:17.356310 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:17.356281 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kjcwb_c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5/dns-node-resolver/0.log" Nov 02 13:57:17.816539 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:17.816502 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:17.816698 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:17.816544 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:17.816698 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:17.816580 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:17.816698 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:17.816645 4431 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:17.816698 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:17.816678 4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:17.816858 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:17.816714 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.816699343 +0000 UTC m=+4.853804253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:17.816858 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:17.816731 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.816724373 +0000 UTC m=+4.853829282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:17.816858 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:17.816712 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:17.816858 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:17.816850 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:19.816833961 +0000 UTC m=+4.853938920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : secret "metrics-daemon-secret" not found Nov 02 13:57:18.560053 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:18.560013 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dtc82_a3206ad3-d2a7-4eec-81cb-db3a09dee305/node-ca/0.log" Nov 02 13:57:19.157814 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:19.157753 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4q2dh_b9d5b255-ab45-4fce-9226-a02f7ffec197/serve-healthcheck-canary/0.log" Nov 02 13:57:19.760225 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:19.760192 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4xbqm_6efd880e-a7eb-4c81-b081-f3a2ca09e1c5/migrator/0.log" Nov 02 13:57:19.832160 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:19.832125 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:19.832300 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:19.832175 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:19.832300 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:19.832201 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:19.832300 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:19.832263 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:19.832300 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:19.832292 4431 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:19.832500 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:19.832322 4431 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 13:57:19.832500 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:19.832332 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.832317952 +0000 UTC m=+8.869422862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : secret "metrics-daemon-secret" not found Nov 02 13:57:19.832500 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:19.832403 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.832383931 +0000 UTC m=+8.869488841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:19.832500 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:19.832417 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls podName:ef68447e-3ad3-4dd2-b353-4aa00584d583 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:23.832409758 +0000 UTC m=+8.869514667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls") pod "dns-default-bv8r5" (UID: "ef68447e-3ad3-4dd2-b353-4aa00584d583") : secret "dns-default-metrics-tls" not found Nov 02 13:57:20.032920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:20.032862 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:23.445517 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:23.445478 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:23.854201 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:23.854117 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:23.854201 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:23.854158 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:23.854405 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:23.854261 4431 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 13:57:23.854405 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:23.854279 4431 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 13:57:23.854405 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:23.854303 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:23.854405 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:23.854320 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs podName:7ba6035a-7b2a-46ab-9506-8fd9b3d369a0 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:31.854305325 +0000 UTC m=+16.891410234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs") pod "network-metrics-daemon-bjbp9" (UID: "7ba6035a-7b2a-46ab-9506-8fd9b3d369a0") : secret "metrics-daemon-secret" not found Nov 02 13:57:23.854405 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:23.854371 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls podName:166c406a-eb7f-41ce-9d49-71f3fda95c35 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:31.854352275 +0000 UTC m=+16.891457188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vwqt2" (UID: "166c406a-eb7f-41ce-9d49-71f3fda95c35") : secret "insights-runtime-extractor-tls" not found Nov 02 13:57:23.854673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:23.854651 4431 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 13:57:23.856924 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:23.856905 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef68447e-3ad3-4dd2-b353-4aa00584d583-metrics-tls\") pod \"dns-default-bv8r5\" (UID: \"ef68447e-3ad3-4dd2-b353-4aa00584d583\") " pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:23.871957 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:23.871925 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:24.171190 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:24.171159 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bv8r5"] Nov 02 13:57:24.175248 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:24.175213 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef68447e_3ad3_4dd2_b353_4aa00584d583.slice/crio-906de802236157e888826d5e1d6a9bf8882ac21401fa26ca99d83a08ced85d66 WatchSource:0}: Error finding container 906de802236157e888826d5e1d6a9bf8882ac21401fa26ca99d83a08ced85d66: Status 404 returned error can't find the container with id 906de802236157e888826d5e1d6a9bf8882ac21401fa26ca99d83a08ced85d66 Nov 02 13:57:24.177161 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:24.177142 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 13:57:24.632971 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:24.632889 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:24.633400 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:24.633122 4431 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 13:57:25.124092 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:25.124032 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bv8r5" event={"ID":"ef68447e-3ad3-4dd2-b353-4aa00584d583","Type":"ContainerStarted","Data":"906de802236157e888826d5e1d6a9bf8882ac21401fa26ca99d83a08ced85d66"} Nov 02 13:57:25.528958 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:25.528925 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5q6hs" Nov 02 13:57:26.128870 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:26.128830 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bv8r5" event={"ID":"ef68447e-3ad3-4dd2-b353-4aa00584d583","Type":"ContainerStarted","Data":"293240ce3311375e33be3214b5f52e69b827c016e6330f387e1673e975094193"} Nov 02 13:57:26.129264 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:26.128927 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bv8r5" event={"ID":"ef68447e-3ad3-4dd2-b353-4aa00584d583","Type":"ContainerStarted","Data":"c66571a64a4445c4c7388481a735c87f99ecaca18b534d9669bc0f8265085020"} Nov 02 13:57:26.129264 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:26.129052 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:31.910311 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:31.910261 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:31.910311 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:31.910315 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:31.912948 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:31.912919 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba6035a-7b2a-46ab-9506-8fd9b3d369a0-metrics-certs\") pod \"network-metrics-daemon-bjbp9\" (UID: \"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0\") " pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:31.913095 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:31.912948 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/166c406a-eb7f-41ce-9d49-71f3fda95c35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vwqt2\" (UID: \"166c406a-eb7f-41ce-9d49-71f3fda95c35\") " pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:31.970327 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:31.970300 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bjbp9" Nov 02 13:57:31.972179 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:31.972160 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vwqt2" Nov 02 13:57:32.302449 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.302377 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bjbp9"] Nov 02 13:57:32.305800 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:32.305764 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba6035a_7b2a_46ab_9506_8fd9b3d369a0.slice/crio-0b07277bd06288f152de1f9f5276249dfc364458dab571063ce0fda398b5c9d4 WatchSource:0}: Error finding container 0b07277bd06288f152de1f9f5276249dfc364458dab571063ce0fda398b5c9d4: Status 404 returned error can't find the container with id 0b07277bd06288f152de1f9f5276249dfc364458dab571063ce0fda398b5c9d4 Nov 02 13:57:32.315021 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:32.315001 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166c406a_eb7f_41ce_9d49_71f3fda95c35.slice/crio-449d277c269d8c4c528fde2a585b0e0dd6fbb746d364f38f0264261bdd277a17 WatchSource:0}: Error finding container 449d277c269d8c4c528fde2a585b0e0dd6fbb746d364f38f0264261bdd277a17: Status 404 returned error can't find the container with id 449d277c269d8c4c528fde2a585b0e0dd6fbb746d364f38f0264261bdd277a17 Nov 02 13:57:32.315440 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.315408 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vwqt2"] Nov 02 13:57:32.666426 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.666387 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff"] Nov 02 13:57:32.668206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.668189 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-t9b9x"] Nov 02 13:57:32.668323 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.668305 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" Nov 02 13:57:32.669804 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.669766 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:32.692263 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.692242 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 13:57:32.692589 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.692558 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gw62f\"" Nov 02 13:57:32.704237 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.704218 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 13:57:32.704336 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.704302 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-9x8fs\"" Nov 02 13:57:32.704336 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.704303 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 13:57:32.704999 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.704978 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff"] Nov 02 13:57:32.705629 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.705612 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-t9b9x"] Nov 02 13:57:32.818553 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.818517 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82fk\" (UniqueName: \"kubernetes.io/projected/2f686f4c-ccfa-4a39-a9c3-a26794765b2b-kube-api-access-b82fk\") pod \"downloads-6dd789c8b7-t9b9x\" (UID: \"2f686f4c-ccfa-4a39-a9c3-a26794765b2b\") " pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:32.818719 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.818590 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f686f4c-ccfa-4a39-a9c3-a26794765b2b-tmp\") pod \"downloads-6dd789c8b7-t9b9x\" (UID: \"2f686f4c-ccfa-4a39-a9c3-a26794765b2b\") " pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:32.818719 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.818625 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f26a21ee-590a-451c-b9fa-5d4f87d61c2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mngff\" (UID: \"f26a21ee-590a-451c-b9fa-5d4f87d61c2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" Nov 02 13:57:32.919150 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.919088 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f26a21ee-590a-451c-b9fa-5d4f87d61c2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mngff\" (UID: \"f26a21ee-590a-451c-b9fa-5d4f87d61c2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" Nov 02 13:57:32.919150 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.919134 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82fk\" (UniqueName: \"kubernetes.io/projected/2f686f4c-ccfa-4a39-a9c3-a26794765b2b-kube-api-access-b82fk\") pod \"downloads-6dd789c8b7-t9b9x\" (UID: \"2f686f4c-ccfa-4a39-a9c3-a26794765b2b\") " pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:32.920058 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.919194 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f686f4c-ccfa-4a39-a9c3-a26794765b2b-tmp\") pod \"downloads-6dd789c8b7-t9b9x\" (UID: \"2f686f4c-ccfa-4a39-a9c3-a26794765b2b\") " pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:32.920058 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.919564 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f686f4c-ccfa-4a39-a9c3-a26794765b2b-tmp\") pod \"downloads-6dd789c8b7-t9b9x\" (UID: \"2f686f4c-ccfa-4a39-a9c3-a26794765b2b\") " pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:32.921905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.921883 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f26a21ee-590a-451c-b9fa-5d4f87d61c2a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mngff\" (UID: \"f26a21ee-590a-451c-b9fa-5d4f87d61c2a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" Nov 02 13:57:32.947177 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.947149 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82fk\" (UniqueName: \"kubernetes.io/projected/2f686f4c-ccfa-4a39-a9c3-a26794765b2b-kube-api-access-b82fk\") pod \"downloads-6dd789c8b7-t9b9x\" (UID: \"2f686f4c-ccfa-4a39-a9c3-a26794765b2b\") " pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:32.978961 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.978936 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" Nov 02 13:57:32.984052 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:32.984028 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:33.148352 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.148313 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vwqt2" event={"ID":"166c406a-eb7f-41ce-9d49-71f3fda95c35","Type":"ContainerStarted","Data":"dbe9019132225646dbbe3d1e8a5e8c1f55381e0a3a1be92970f934b31dbb08b1"} Nov 02 13:57:33.148524 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.148388 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vwqt2" event={"ID":"166c406a-eb7f-41ce-9d49-71f3fda95c35","Type":"ContainerStarted","Data":"e67b1634b998026e713ae2fc3958622c818f3903adf87fc64f9689b2db30b101"} Nov 02 13:57:33.148524 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.148418 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vwqt2" event={"ID":"166c406a-eb7f-41ce-9d49-71f3fda95c35","Type":"ContainerStarted","Data":"449d277c269d8c4c528fde2a585b0e0dd6fbb746d364f38f0264261bdd277a17"} Nov 02 13:57:33.149629 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.149601 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjbp9" event={"ID":"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0","Type":"ContainerStarted","Data":"0b07277bd06288f152de1f9f5276249dfc364458dab571063ce0fda398b5c9d4"} Nov 02 13:57:33.279299 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.279274 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff"] Nov 02 13:57:33.285749 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:33.285720 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26a21ee_590a_451c_b9fa_5d4f87d61c2a.slice/crio-a4063c0f989ae03683d1fbf730adfeb278972afb5aa0b60e59ad68115b47243d WatchSource:0}: Error finding container a4063c0f989ae03683d1fbf730adfeb278972afb5aa0b60e59ad68115b47243d: Status 404 returned error can't find the container with id a4063c0f989ae03683d1fbf730adfeb278972afb5aa0b60e59ad68115b47243d Nov 02 13:57:33.295001 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.294935 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-t9b9x"] Nov 02 13:57:33.918825 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.918509 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-54f8d7fffb-8lntm"] Nov 02 13:57:33.922823 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.921912 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:33.941812 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.941791 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 13:57:33.955432 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.955408 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f8d7fffb-8lntm"] Nov 02 13:57:33.955552 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.955538 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bwlsh\"" Nov 02 13:57:33.955611 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.955593 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 13:57:33.955665 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.955646 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 13:57:33.955744 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.955730 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 13:57:33.955828 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:33.955810 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 13:57:34.028392 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.028358 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7m8l\" (UniqueName: \"kubernetes.io/projected/f5d73de5-3e17-4071-b707-2c12734ed196-kube-api-access-s7m8l\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.028561 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.028399 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-oauth-config\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.028561 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.028432 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-oauth-serving-cert\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.028683 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.028567 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-console-config\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.028683 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.028601 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-service-ca\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.028683 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.028655 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5d73de5-3e17-4071-b707-2c12734ed196-tmp\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.028868 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.028696 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-serving-cert\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.129589 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.129552 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-serving-cert\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.129589 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.129603 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7m8l\" (UniqueName: \"kubernetes.io/projected/f5d73de5-3e17-4071-b707-2c12734ed196-kube-api-access-s7m8l\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.129852 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.129632 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-oauth-config\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.129852 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.129662 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-oauth-serving-cert\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.129852 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.129698 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-console-config\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.129852 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.129722 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-service-ca\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.129852 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.129764 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5d73de5-3e17-4071-b707-2c12734ed196-tmp\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.131433 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.130163 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5d73de5-3e17-4071-b707-2c12734ed196-tmp\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.131433 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.130904 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-oauth-serving-cert\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.131433 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.131367 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-console-config\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.131689 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.131669 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-service-ca\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.134494 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.134471 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-serving-cert\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.149274 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.149252 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-oauth-config\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.156882 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.156761 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjbp9" event={"ID":"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0","Type":"ContainerStarted","Data":"be612d6c3826ef18437755b4b55284d2964a557af467efc3878304f2e553e0a6"} Nov 02 13:57:34.156882 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.156841 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bjbp9" event={"ID":"7ba6035a-7b2a-46ab-9506-8fd9b3d369a0","Type":"ContainerStarted","Data":"debc4bba713d29b382275c5fe18b942c024d38d41f5146036ee522cc186a3439"} Nov 02 13:57:34.157794 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.157735 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7m8l\" (UniqueName: \"kubernetes.io/projected/f5d73de5-3e17-4071-b707-2c12734ed196-kube-api-access-s7m8l\") pod \"console-54f8d7fffb-8lntm\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.158879 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.158808 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-t9b9x" event={"ID":"2f686f4c-ccfa-4a39-a9c3-a26794765b2b","Type":"ContainerStarted","Data":"b50277611b1fd9a455138f625268a78710d7c13070ae8e50892c6d796098dc41"} Nov 02 13:57:34.160628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.160573 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" event={"ID":"f26a21ee-590a-451c-b9fa-5d4f87d61c2a","Type":"ContainerStarted","Data":"a4063c0f989ae03683d1fbf730adfeb278972afb5aa0b60e59ad68115b47243d"} Nov 02 13:57:34.239267 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.239204 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:34.553150 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:34.551949 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f8d7fffb-8lntm"] Nov 02 13:57:34.556110 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:34.556086 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d73de5_3e17_4071_b707_2c12734ed196.slice/crio-9f42158723c57de4170a4792b7b31ce22613dc371796bb53f55b840979d4a948 WatchSource:0}: Error finding container 9f42158723c57de4170a4792b7b31ce22613dc371796bb53f55b840979d4a948: Status 404 returned error can't find the container with id 9f42158723c57de4170a4792b7b31ce22613dc371796bb53f55b840979d4a948 Nov 02 13:57:35.166491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.166456 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vwqt2" event={"ID":"166c406a-eb7f-41ce-9d49-71f3fda95c35","Type":"ContainerStarted","Data":"ad68800b17329c245f5c309756039e186f6e462988ee23eaf602c249d2650728"} Nov 02 13:57:35.168452 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.168418 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" event={"ID":"f26a21ee-590a-451c-b9fa-5d4f87d61c2a","Type":"ContainerStarted","Data":"2b590939124730a962d48d24fb14566583d2b2382369987397dcf0a754b921f9"} Nov 02 13:57:35.168704 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.168689 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" Nov 02 13:57:35.170419 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.170395 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f8d7fffb-8lntm" event={"ID":"f5d73de5-3e17-4071-b707-2c12734ed196","Type":"ContainerStarted","Data":"9f42158723c57de4170a4792b7b31ce22613dc371796bb53f55b840979d4a948"} Nov 02 13:57:35.174675 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.174657 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" Nov 02 13:57:35.263296 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.263240 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mngff" podStartSLOduration=2.221865333 podStartE2EDuration="3.263224065s" totalImagesPullingTime="1.041358732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:35.263180148 +0000 UTC m=+20.300285114" watchObservedRunningTime="2026-11-02 13:57:35.263224065 +0000 UTC m=+20.300328997" Nov 02 13:57:35.627323 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.627254 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nv82m"] Nov 02 13:57:35.630653 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.630631 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.650081 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.650057 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 13:57:35.650332 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.650299 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 13:57:35.650437 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.650340 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 13:57:35.650563 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.650524 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 13:57:35.650632 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.650569 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2r447\"" Nov 02 13:57:35.650632 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.650600 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 13:57:35.663287 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.663262 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nv82m"] Nov 02 13:57:35.740091 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.740052 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d022dc5-98a1-4834-8fd9-749f05411719-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.740216 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.740109 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqtbq\" (UniqueName: \"kubernetes.io/projected/9d022dc5-98a1-4834-8fd9-749f05411719-kube-api-access-gqtbq\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.740216 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.740139 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d022dc5-98a1-4834-8fd9-749f05411719-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.740216 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.740177 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d022dc5-98a1-4834-8fd9-749f05411719-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.842411 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.840857 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d022dc5-98a1-4834-8fd9-749f05411719-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.842411 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.840921 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqtbq\" (UniqueName: \"kubernetes.io/projected/9d022dc5-98a1-4834-8fd9-749f05411719-kube-api-access-gqtbq\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.842411 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.840952 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d022dc5-98a1-4834-8fd9-749f05411719-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.842411 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.840992 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d022dc5-98a1-4834-8fd9-749f05411719-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.842411 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.841765 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d022dc5-98a1-4834-8fd9-749f05411719-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.845062 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.845035 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d022dc5-98a1-4834-8fd9-749f05411719-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.849396 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.849373 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d022dc5-98a1-4834-8fd9-749f05411719-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.868426 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.868403 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqtbq\" (UniqueName: \"kubernetes.io/projected/9d022dc5-98a1-4834-8fd9-749f05411719-kube-api-access-gqtbq\") pod \"prometheus-operator-9cd89b74c-nv82m\" (UID: \"9d022dc5-98a1-4834-8fd9-749f05411719\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:35.946378 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:35.946346 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" Nov 02 13:57:36.133656 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:36.133610 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bv8r5" Nov 02 13:57:36.255986 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:36.255948 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nv82m"] Nov 02 13:57:36.257468 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:36.257419 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d022dc5_98a1_4834_8fd9_749f05411719.slice/crio-e3becae0f357c4b04ab098dcc3a0549d967ea23a27ec5947cd119a4a28677f18 WatchSource:0}: Error finding container e3becae0f357c4b04ab098dcc3a0549d967ea23a27ec5947cd119a4a28677f18: Status 404 returned error can't find the container with id e3becae0f357c4b04ab098dcc3a0549d967ea23a27ec5947cd119a4a28677f18 Nov 02 13:57:37.180886 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:37.180855 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" event={"ID":"9d022dc5-98a1-4834-8fd9-749f05411719","Type":"ContainerStarted","Data":"e3becae0f357c4b04ab098dcc3a0549d967ea23a27ec5947cd119a4a28677f18"} Nov 02 13:57:38.186199 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:38.186156 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" event={"ID":"9d022dc5-98a1-4834-8fd9-749f05411719","Type":"ContainerStarted","Data":"0259a92e65b59d67e88a29c27fc276fbcda2ca3b720b5a0fe5f5d7c7248f93b7"} Nov 02 13:57:38.186692 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:38.186227 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" event={"ID":"9d022dc5-98a1-4834-8fd9-749f05411719","Type":"ContainerStarted","Data":"cc1f872e88dcf68adf0269c165e55220d36b32f646941bc0474def65ca10ce00"} Nov 02 13:57:38.188281 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:38.188255 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f8d7fffb-8lntm" event={"ID":"f5d73de5-3e17-4071-b707-2c12734ed196","Type":"ContainerStarted","Data":"d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc"} Nov 02 13:57:38.233032 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:38.232978 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nv82m" podStartSLOduration=1.69952812 podStartE2EDuration="3.232960917s" totalImagesPullingTime="1.533432797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:38.231563007 +0000 UTC m=+23.268667940" watchObservedRunningTime="2026-11-02 13:57:38.232960917 +0000 UTC m=+23.270065851" Nov 02 13:57:40.230602 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.230550 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-54f8d7fffb-8lntm" podStartSLOduration=3.993944386 podStartE2EDuration="7.230533928s" totalImagesPullingTime="3.236589542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:38.290018546 +0000 UTC m=+23.327123479" watchObservedRunningTime="2026-11-02 13:57:40.230533928 +0000 UTC m=+25.267638864" Nov 02 13:57:40.231191 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.231137 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd"] Nov 02 13:57:40.236070 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.236045 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.270049 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.270029 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 13:57:40.270174 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.270067 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8frsk\"" Nov 02 13:57:40.270219 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.270169 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 13:57:40.271501 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.271484 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd"] Nov 02 13:57:40.273323 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.273301 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/197100a3-c701-4a29-a7de-de6df973e373-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.273439 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.273364 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/197100a3-c701-4a29-a7de-de6df973e373-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.273501 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.273443 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/197100a3-c701-4a29-a7de-de6df973e373-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.273551 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.273524 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrjk\" (UniqueName: \"kubernetes.io/projected/197100a3-c701-4a29-a7de-de6df973e373-kube-api-access-dcrjk\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.289328 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.289304 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9jmvm"] Nov 02 13:57:40.292630 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.292609 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.297861 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.297839 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2"] Nov 02 13:57:40.301196 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.301178 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.328241 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.328182 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2"] Nov 02 13:57:40.357684 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.357663 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 13:57:40.357916 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.357874 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 13:57:40.358089 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.358071 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 13:57:40.358163 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.358099 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 13:57:40.358163 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.358107 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z8rpr\"" Nov 02 13:57:40.358163 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.358127 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xnscf\"" Nov 02 13:57:40.374326 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374306 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-root\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.374425 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374336 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.374425 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374369 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/197100a3-c701-4a29-a7de-de6df973e373-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.374538 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374488 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/197100a3-c701-4a29-a7de-de6df973e373-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.374615 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374598 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.374671 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374632 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.374671 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374664 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrjk\" (UniqueName: \"kubernetes.io/projected/197100a3-c701-4a29-a7de-de6df973e373-kube-api-access-dcrjk\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.374801 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374733 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-textfile\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.374801 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374761 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4b22\" (UniqueName: \"kubernetes.io/projected/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-kube-api-access-q4b22\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.374905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374860 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-sys\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.374905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374897 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.374998 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374930 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-tls\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.374998 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.374961 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwc6r\" (UniqueName: \"kubernetes.io/projected/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-api-access-hwc6r\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.375065 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.375039 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8088b390-9d13-4cfa-903a-68ac3f2a81a4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.375104 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.375077 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.375156 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.375118 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/197100a3-c701-4a29-a7de-de6df973e373-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.375211 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.375174 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8088b390-9d13-4cfa-903a-68ac3f2a81a4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.375264 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.375220 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-wtmp\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.375264 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.375252 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-metrics-client-ca\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.375760 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.375736 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/197100a3-c701-4a29-a7de-de6df973e373-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.376864 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.376843 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/197100a3-c701-4a29-a7de-de6df973e373-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.376994 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.376978 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/197100a3-c701-4a29-a7de-de6df973e373-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.400286 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.400248 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 13:57:40.400528 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.400503 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 13:57:40.400649 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.400626 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrjk\" (UniqueName: \"kubernetes.io/projected/197100a3-c701-4a29-a7de-de6df973e373-kube-api-access-dcrjk\") pod \"openshift-state-metrics-7bd7489957-s99vd\" (UID: \"197100a3-c701-4a29-a7de-de6df973e373\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.475901 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.475874 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwc6r\" (UniqueName: \"kubernetes.io/projected/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-api-access-hwc6r\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.476026 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.475917 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8088b390-9d13-4cfa-903a-68ac3f2a81a4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.476026 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.475946 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.476026 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.475986 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8088b390-9d13-4cfa-903a-68ac3f2a81a4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.476026 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476022 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-wtmp\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476241 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476045 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-metrics-client-ca\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476241 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476075 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-root\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476241 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476103 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476241 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476141 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.476241 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476168 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.476241 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476199 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-textfile\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476241 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476228 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4b22\" (UniqueName: \"kubernetes.io/projected/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-kube-api-access-q4b22\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476569 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476253 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-sys\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476569 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476266 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-wtmp\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476569 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476278 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476569 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476306 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-tls\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.476569 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.476523 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8088b390-9d13-4cfa-903a-68ac3f2a81a4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.477092 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.477066 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.478484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.477566 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8088b390-9d13-4cfa-903a-68ac3f2a81a4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.478484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.477633 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-root\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.478484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.477680 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-textfile\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.478484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.477979 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-sys\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.478484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.478025 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-metrics-client-ca\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.478484 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.478441 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.478911 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.478888 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-tls\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.479066 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.479046 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.480158 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.480135 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.480399 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.480380 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.501628 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.501580 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwc6r\" (UniqueName: \"kubernetes.io/projected/8088b390-9d13-4cfa-903a-68ac3f2a81a4-kube-api-access-hwc6r\") pod \"kube-state-metrics-5dc8d7d89b-rwvx2\" (UID: \"8088b390-9d13-4cfa-903a-68ac3f2a81a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.502329 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.502308 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4b22\" (UniqueName: \"kubernetes.io/projected/361c7df3-61c9-4cdb-8c3b-4d59fba75b1d-kube-api-access-q4b22\") pod \"node-exporter-9jmvm\" (UID: \"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d\") " pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.553482 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.553460 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" Nov 02 13:57:40.610872 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.610741 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9jmvm" Nov 02 13:57:40.619199 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.619171 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" Nov 02 13:57:40.634870 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:40.634843 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361c7df3_61c9_4cdb_8c3b_4d59fba75b1d.slice/crio-2f757c26f60308fd573bac21397a03a0601b63890d357f0c32c7f68978b167b5 WatchSource:0}: Error finding container 2f757c26f60308fd573bac21397a03a0601b63890d357f0c32c7f68978b167b5: Status 404 returned error can't find the container with id 2f757c26f60308fd573bac21397a03a0601b63890d357f0c32c7f68978b167b5 Nov 02 13:57:40.877926 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.877830 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd"] Nov 02 13:57:40.881869 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:40.881840 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197100a3_c701_4a29_a7de_de6df973e373.slice/crio-9e8ae6cbd374958abddc1fcd0eab55f1526cc1755c6e4be2e110864f372b1790 WatchSource:0}: Error finding container 9e8ae6cbd374958abddc1fcd0eab55f1526cc1755c6e4be2e110864f372b1790: Status 404 returned error can't find the container with id 9e8ae6cbd374958abddc1fcd0eab55f1526cc1755c6e4be2e110864f372b1790 Nov 02 13:57:40.930643 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:40.930619 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2"] Nov 02 13:57:40.933384 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:40.933359 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8088b390_9d13_4cfa_903a_68ac3f2a81a4.slice/crio-988a33ac405af759da8cf3cd52f866731a511aaaa72e26b85032d3c169d6af55 WatchSource:0}: Error finding container 988a33ac405af759da8cf3cd52f866731a511aaaa72e26b85032d3c169d6af55: Status 404 returned error can't find the container with id 988a33ac405af759da8cf3cd52f866731a511aaaa72e26b85032d3c169d6af55 Nov 02 13:57:41.201048 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.201014 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9jmvm" event={"ID":"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d","Type":"ContainerStarted","Data":"2f757c26f60308fd573bac21397a03a0601b63890d357f0c32c7f68978b167b5"} Nov 02 13:57:41.203247 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.203122 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" event={"ID":"197100a3-c701-4a29-a7de-de6df973e373","Type":"ContainerStarted","Data":"ace7d57eb386da5cb369801a2934c66513dc30babf7e137d26ccf08380e30c48"} Nov 02 13:57:41.203247 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.203183 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" event={"ID":"197100a3-c701-4a29-a7de-de6df973e373","Type":"ContainerStarted","Data":"c9a632c9278e632f524775ebb00d8a430660c42d78f20b9204a9b8888d89c0f9"} Nov 02 13:57:41.203247 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.203212 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" event={"ID":"197100a3-c701-4a29-a7de-de6df973e373","Type":"ContainerStarted","Data":"9e8ae6cbd374958abddc1fcd0eab55f1526cc1755c6e4be2e110864f372b1790"} Nov 02 13:57:41.204503 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.204480 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" event={"ID":"8088b390-9d13-4cfa-903a-68ac3f2a81a4","Type":"ContainerStarted","Data":"988a33ac405af759da8cf3cd52f866731a511aaaa72e26b85032d3c169d6af55"} Nov 02 13:57:41.512262 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.511061 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zx5gw" Nov 02 13:57:41.772911 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.772889 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:57:41.777059 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.777036 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.792436 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792416 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-web-config\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.792557 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792451 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.792557 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792472 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.792557 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792537 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.792811 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792584 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.792811 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792606 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-config-volume\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.792811 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792678 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-config-out\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.792811 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792722 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxd2g\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-kube-api-access-pxd2g\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.793037 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792854 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.793037 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792900 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.793037 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.792924 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.793037 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.793023 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.793231 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.793063 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.795922 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.795375 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 13:57:41.795922 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.795422 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 13:57:41.795922 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.795460 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 13:57:41.795922 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.795509 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 13:57:41.795922 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.795577 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 13:57:41.795922 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.795681 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bgfwj\"" Nov 02 13:57:41.795922 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.795427 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 13:57:41.795922 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.795880 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 13:57:41.796363 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.796083 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 13:57:41.812206 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.811940 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 13:57:41.823438 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.823414 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894296 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894341 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894363 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894388 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894413 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-config-volume\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894438 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-config-out\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894461 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxd2g\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-kube-api-access-pxd2g\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894482 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894511 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894531 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894561 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894594 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.895011 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.894628 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-web-config\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.898181 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.896922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.898181 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.897446 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-web-config\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.898181 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.897581 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.898181 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.898156 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.898617 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.898591 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.899144 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.899103 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.899480 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.899435 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.900007 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.899977 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.900669 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.900621 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.901437 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.901411 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-config-out\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.901520 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.901472 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-config-volume\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.902715 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.902695 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:41.923103 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:41.923063 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxd2g\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-kube-api-access-pxd2g\") pod \"alertmanager-main-0\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:42.106515 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:42.106435 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:57:42.210614 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:42.210582 4431 generic.go:356] "Generic (PLEG): container finished" podID="361c7df3-61c9-4cdb-8c3b-4d59fba75b1d" containerID="2c9229a1d1bb39328034ec590ce667bb536be54b760c329225836caae2ab42c8" exitCode=0 Nov 02 13:57:42.210753 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:42.210667 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9jmvm" event={"ID":"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d","Type":"ContainerDied","Data":"2c9229a1d1bb39328034ec590ce667bb536be54b760c329225836caae2ab42c8"} Nov 02 13:57:42.453744 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:42.453713 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:57:42.458263 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:42.458221 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18649210_246f_4218_8aa8_fb0f4c80095e.slice/crio-f2b42363899f427f0e23367e99595ba3d962674e7878f42e9308f9f45f27b840 WatchSource:0}: Error finding container f2b42363899f427f0e23367e99595ba3d962674e7878f42e9308f9f45f27b840: Status 404 returned error can't find the container with id f2b42363899f427f0e23367e99595ba3d962674e7878f42e9308f9f45f27b840 Nov 02 13:57:43.218080 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.217939 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" event={"ID":"8088b390-9d13-4cfa-903a-68ac3f2a81a4","Type":"ContainerStarted","Data":"760abcde711334b5ed3a828a3f949afbe169b955c8440bfdce07cca9f1f40cc9"} Nov 02 13:57:43.218080 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.218007 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" event={"ID":"8088b390-9d13-4cfa-903a-68ac3f2a81a4","Type":"ContainerStarted","Data":"ca8badf9ac07cceaccc277551cf1d6d458772e2daae542620f41701535999773"} Nov 02 13:57:43.218080 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.218037 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" event={"ID":"8088b390-9d13-4cfa-903a-68ac3f2a81a4","Type":"ContainerStarted","Data":"3f11bcd84a9cdb5ec54dd855945a4f3ae03a774a49f0b59bf29a14e22766b95d"} Nov 02 13:57:43.219669 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.219635 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerStarted","Data":"f2b42363899f427f0e23367e99595ba3d962674e7878f42e9308f9f45f27b840"} Nov 02 13:57:43.222575 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.222545 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9jmvm" event={"ID":"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d","Type":"ContainerStarted","Data":"5bd28c495cf1ac622245f986bbdf2e7e4deb0d4e68ca277afdb1f24c635646c4"} Nov 02 13:57:43.222673 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.222602 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9jmvm" event={"ID":"361c7df3-61c9-4cdb-8c3b-4d59fba75b1d","Type":"ContainerStarted","Data":"ffdf0ef350c64f9a6eab4baa71483f905a120404cc1b11c603e7bfb7f5535f9f"} Nov 02 13:57:43.225178 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.225124 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" event={"ID":"197100a3-c701-4a29-a7de-de6df973e373","Type":"ContainerStarted","Data":"969d18ddd830d0c3776c4fcf4b6331610560a11f1907cbaae48b61a75a725acb"} Nov 02 13:57:43.284986 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.283954 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rwvx2" podStartSLOduration=1.690833619 podStartE2EDuration="3.283937042s" totalImagesPullingTime="1.593103423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:43.28340085 +0000 UTC m=+28.320505776" watchObservedRunningTime="2026-11-02 13:57:43.283937042 +0000 UTC m=+28.321041975" Nov 02 13:57:43.357938 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.357870 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-s99vd" podStartSLOduration=2.090965441 podStartE2EDuration="3.35785101s" totalImagesPullingTime="1.266885569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:43.356835075 +0000 UTC m=+28.393940008" watchObservedRunningTime="2026-11-02 13:57:43.35785101 +0000 UTC m=+28.394955978" Nov 02 13:57:43.414358 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.413348 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9jmvm" podStartSLOduration=2.257478605 podStartE2EDuration="3.413330486s" totalImagesPullingTime="1.067748708s" totalInitContainerRuntime="88.103173ms" isStatefulPod=true podCreationTimestamp="2026-11-02 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:43.412872035 +0000 UTC m=+28.449976969" watchObservedRunningTime="2026-11-02 13:57:43.413330486 +0000 UTC m=+28.450435419" Nov 02 13:57:43.449013 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:43.448982 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tqb8d" Nov 02 13:57:44.230686 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:44.230645 4431 generic.go:356] "Generic (PLEG): container finished" podID="18649210-246f-4218-8aa8-fb0f4c80095e" containerID="1cb32360504b3752f6b285df414c6c58f51c5bae346e2843f8b62851b9a4f9fd" exitCode=0 Nov 02 13:57:44.231173 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:44.230747 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerDied","Data":"1cb32360504b3752f6b285df414c6c58f51c5bae346e2843f8b62851b9a4f9fd"} Nov 02 13:57:44.240349 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:44.240326 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:44.240465 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:44.240380 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:44.246407 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:44.246383 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:45.239597 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:45.239568 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:57:46.242196 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:46.242075 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerStarted","Data":"3f28c2e7975088c0c85e6e9ab7acb15ff6e11d2e75aaaad961734d362c0c6602"} Nov 02 13:57:46.922905 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:46.922873 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-574cbdb96f-cg9k7"] Nov 02 13:57:46.939419 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:46.939390 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:46.949952 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:46.949920 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574cbdb96f-cg9k7"] Nov 02 13:57:46.985115 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:46.985086 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 13:57:47.042414 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.042382 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-service-ca\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.042560 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.042489 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szrt\" (UniqueName: \"kubernetes.io/projected/ef25160b-7ff9-42e3-9ba2-af126544bac3-kube-api-access-7szrt\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.042560 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.042520 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-trusted-ca-bundle\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.042560 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.042542 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-oauth-config\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.042715 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.042563 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-config\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.042715 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.042628 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-serving-cert\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.042843 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.042743 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-oauth-serving-cert\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.042843 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.042792 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef25160b-7ff9-42e3-9ba2-af126544bac3-tmp\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144022 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.143912 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szrt\" (UniqueName: \"kubernetes.io/projected/ef25160b-7ff9-42e3-9ba2-af126544bac3-kube-api-access-7szrt\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144022 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.143960 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-trusted-ca-bundle\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144022 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.143994 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-oauth-config\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144022 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.144026 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-config\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144308 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.144053 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-serving-cert\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144308 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.144090 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-oauth-serving-cert\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144308 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.144118 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef25160b-7ff9-42e3-9ba2-af126544bac3-tmp\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144308 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.144169 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-service-ca\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.144956 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.144928 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-service-ca\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.146159 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.145957 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-trusted-ca-bundle\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.146510 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.146456 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-config\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.146510 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.146505 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-oauth-serving-cert\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.146700 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.146615 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef25160b-7ff9-42e3-9ba2-af126544bac3-tmp\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.148709 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.148689 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-serving-cert\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.154318 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.154296 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-oauth-config\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.189974 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.189927 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szrt\" (UniqueName: \"kubernetes.io/projected/ef25160b-7ff9-42e3-9ba2-af126544bac3-kube-api-access-7szrt\") pod \"console-574cbdb96f-cg9k7\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.249924 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.249889 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerStarted","Data":"353a7c61c845265ce519869588cc41db67bb81db679ccb6491c8d4b92737141d"} Nov 02 13:57:47.250312 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.249976 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerStarted","Data":"f743a84796ba585390512a199a30466a3952b620827f02944580b1ee1c51aa85"} Nov 02 13:57:47.250312 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.250014 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerStarted","Data":"b190ab69b25c5abea4b30ac71425ede143738a88dc5495711968e4489e0156ab"} Nov 02 13:57:47.250312 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.250048 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerStarted","Data":"f4d454c5cd40889173fd7b41d8e0e7f3dba1e87ae568c0d294a7db799414814f"} Nov 02 13:57:47.264248 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.264069 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:47.533713 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.533632 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:57:47.550546 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.550508 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.568431 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.568408 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 13:57:47.583514 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583482 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 13:57:47.583653 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583542 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 13:57:47.583653 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583561 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 13:57:47.583653 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583573 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 13:57:47.583653 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583605 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 13:57:47.583826 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583671 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 13:57:47.583826 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583677 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 13:57:47.583826 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583676 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 13:57:47.583826 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583674 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 13:57:47.583826 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.583732 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 13:57:47.594223 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.594201 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 13:57:47.594223 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.594220 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 13:57:47.594364 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.594224 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 13:57:47.594364 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.594245 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6l2d5\"" Nov 02 13:57:47.594364 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.594273 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-inmegb4reb5t\"" Nov 02 13:57:47.614051 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.613849 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:57:47.623701 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.623669 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574cbdb96f-cg9k7"] Nov 02 13:57:47.650933 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.650903 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-web-config\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651119 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.650945 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-config\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651119 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.650976 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-config-out\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651119 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651031 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651119 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651059 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651119 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651079 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651402 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651132 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651402 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651190 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651402 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651226 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651402 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651256 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651402 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651279 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvjk\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-kube-api-access-spvjk\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651402 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651309 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651402 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651373 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651726 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651439 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651726 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651495 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651726 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651528 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651726 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651552 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651726 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651589 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651726 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651622 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.651726 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.651637 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.671860 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.671838 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 13:57:47.752761 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.752733 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.752761 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.752763 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.752944 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.752794 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-web-config\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.752944 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.752815 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-config\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.752969 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-config-out\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753006 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753039 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753182 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753098 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753182 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753125 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753182 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753166 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753461 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753717 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753689 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753857 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753714 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753857 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753761 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753857 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753830 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.753857 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753855 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvjk\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-kube-api-access-spvjk\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754077 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753886 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754077 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753915 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754077 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753944 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754077 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.753986 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754077 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.754018 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754077 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.754051 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754369 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.754092 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754369 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.754308 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.754812 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.754741 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.755328 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.755304 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.757981 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.757949 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.758087 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.758010 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.758390 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.758365 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-web-config\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.758390 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.758386 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.758839 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.758816 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.759158 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.759140 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-config-out\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.759403 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.759251 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-config\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.759403 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.759384 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.759403 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.759396 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.759984 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.759960 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.760092 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.759969 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.760355 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.760338 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.761164 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.761146 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.784514 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.784468 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvjk\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-kube-api-access-spvjk\") pod \"prometheus-k8s-0\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:47.788632 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:47.788612 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef25160b_7ff9_42e3_9ba2_af126544bac3.slice/crio-34131a06f159786aaf1f5273042bf10414e1d5b7644706d05575aaa61c207fca WatchSource:0}: Error finding container 34131a06f159786aaf1f5273042bf10414e1d5b7644706d05575aaa61c207fca: Status 404 returned error can't find the container with id 34131a06f159786aaf1f5273042bf10414e1d5b7644706d05575aaa61c207fca Nov 02 13:57:47.983993 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:47.983948 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:57:48.257529 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:48.257493 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerStarted","Data":"ebb10bab3010feff012ec3b3c60486371e1ea3ab1fc6bc4a4eb0a63eb681fb81"} Nov 02 13:57:48.259409 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:48.259386 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574cbdb96f-cg9k7" event={"ID":"ef25160b-7ff9-42e3-9ba2-af126544bac3","Type":"ContainerStarted","Data":"f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3"} Nov 02 13:57:48.259536 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:48.259421 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574cbdb96f-cg9k7" event={"ID":"ef25160b-7ff9-42e3-9ba2-af126544bac3","Type":"ContainerStarted","Data":"34131a06f159786aaf1f5273042bf10414e1d5b7644706d05575aaa61c207fca"} Nov 02 13:57:48.321132 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:48.321054 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.180748739 podStartE2EDuration="7.321040114s" totalImagesPullingTime="4.127523183s" totalInitContainerRuntime="12.768192ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:48.320041686 +0000 UTC m=+33.357146620" watchObservedRunningTime="2026-11-02 13:57:48.321040114 +0000 UTC m=+33.358145047" Nov 02 13:57:48.360562 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:48.359920 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:57:48.363397 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:57:48.363368 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e7a2ac_2ac4_497e_942c_c1b44943876c.slice/crio-51dc8da85bd97520d1580db75784ef2bd7fffebbb092bc8a4092745fa36ddc8d WatchSource:0}: Error finding container 51dc8da85bd97520d1580db75784ef2bd7fffebbb092bc8a4092745fa36ddc8d: Status 404 returned error can't find the container with id 51dc8da85bd97520d1580db75784ef2bd7fffebbb092bc8a4092745fa36ddc8d Nov 02 13:57:48.393195 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:48.393130 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-574cbdb96f-cg9k7" podStartSLOduration=2.393111753 podStartE2EDuration="2.393111753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:48.380677918 +0000 UTC m=+33.417782851" watchObservedRunningTime="2026-11-02 13:57:48.393111753 +0000 UTC m=+33.430216686" Nov 02 13:57:49.265575 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:49.265539 4431 generic.go:356] "Generic (PLEG): container finished" podID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" exitCode=0 Nov 02 13:57:49.266030 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:49.265635 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerDied","Data":"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6"} Nov 02 13:57:49.266030 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:49.265736 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerStarted","Data":"51dc8da85bd97520d1580db75784ef2bd7fffebbb092bc8a4092745fa36ddc8d"} Nov 02 13:57:51.277716 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:51.277481 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-t9b9x" event={"ID":"2f686f4c-ccfa-4a39-a9c3-a26794765b2b","Type":"ContainerStarted","Data":"6bed9f92ecb4d61ee6df9647d9ab1bcc4a9fdda8d112b1d13bc63f30d56bc5ac"} Nov 02 13:57:51.278236 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:51.277960 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:51.288819 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:57:51.288762 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 13:57:51.289817 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:51.289769 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-t9b9x" Nov 02 13:57:51.351862 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:51.351796 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-t9b9x" podStartSLOduration=1.910312116 podStartE2EDuration="19.351766657s" totalImagesPullingTime="17.441454541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:51.350238695 +0000 UTC m=+36.387343627" watchObservedRunningTime="2026-11-02 13:57:51.351766657 +0000 UTC m=+36.388871589" Nov 02 13:57:54.317728 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:54.317695 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerStarted","Data":"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8"} Nov 02 13:57:54.318295 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:54.317761 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerStarted","Data":"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444"} Nov 02 13:57:56.331263 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:56.331224 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerStarted","Data":"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e"} Nov 02 13:57:56.331591 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:56.331346 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerStarted","Data":"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900"} Nov 02 13:57:57.265093 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.265058 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:57.265364 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.265119 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:57.270562 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.270536 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:57.341109 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.341059 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerStarted","Data":"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0"} Nov 02 13:57:57.341505 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.341135 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerStarted","Data":"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8"} Nov 02 13:57:57.345107 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.345082 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:57:57.402751 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.402704 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.798682018 podStartE2EDuration="10.402688685s" totalImagesPullingTime="6.574025196s" totalInitContainerRuntime="29.981471ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:57.401373031 +0000 UTC m=+42.438477975" watchObservedRunningTime="2026-11-02 13:57:57.402688685 +0000 UTC m=+42.439793618" Nov 02 13:57:57.490890 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.490858 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54f8d7fffb-8lntm"] Nov 02 13:57:57.985035 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:57:57.984956 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:21.281052 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:21.281012 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 13:58:22.515648 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.515586 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-54f8d7fffb-8lntm" podUID="f5d73de5-3e17-4071-b707-2c12734ed196" containerName="console" containerID="cri-o://d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc" gracePeriod=15 Nov 02 13:58:22.780919 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.780863 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:58:22.856106 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856078 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-console-config\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-console-config\") pod \"f5d73de5-3e17-4071-b707-2c12734ed196\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " Nov 02 13:58:22.856258 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856116 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-oauth-serving-cert\") pod \"f5d73de5-3e17-4071-b707-2c12734ed196\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " Nov 02 13:58:22.856258 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856139 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d73de5-3e17-4071-b707-2c12734ed196-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5d73de5-3e17-4071-b707-2c12734ed196-tmp\") pod \"f5d73de5-3e17-4071-b707-2c12734ed196\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " Nov 02 13:58:22.856371 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856274 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-oauth-config\") pod \"f5d73de5-3e17-4071-b707-2c12734ed196\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " Nov 02 13:58:22.856371 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856320 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d73de5-3e17-4071-b707-2c12734ed196-kube-api-access-s7m8l\" (UniqueName: \"kubernetes.io/projected/f5d73de5-3e17-4071-b707-2c12734ed196-kube-api-access-s7m8l\") pod \"f5d73de5-3e17-4071-b707-2c12734ed196\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " Nov 02 13:58:22.856475 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856382 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-serving-cert\") pod \"f5d73de5-3e17-4071-b707-2c12734ed196\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " Nov 02 13:58:22.856475 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856423 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-service-ca\") pod \"f5d73de5-3e17-4071-b707-2c12734ed196\" (UID: \"f5d73de5-3e17-4071-b707-2c12734ed196\") " Nov 02 13:58:22.856475 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856440 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d73de5-3e17-4071-b707-2c12734ed196-tmp" pod "f5d73de5-3e17-4071-b707-2c12734ed196" (UID: "f5d73de5-3e17-4071-b707-2c12734ed196"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:58:22.856619 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856541 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-oauth-serving-cert" pod "f5d73de5-3e17-4071-b707-2c12734ed196" (UID: "f5d73de5-3e17-4071-b707-2c12734ed196"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:22.856619 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856560 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-console-config" pod "f5d73de5-3e17-4071-b707-2c12734ed196" (UID: "f5d73de5-3e17-4071-b707-2c12734ed196"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:22.856860 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.856834 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-service-ca" pod "f5d73de5-3e17-4071-b707-2c12734ed196" (UID: "f5d73de5-3e17-4071-b707-2c12734ed196"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:22.857067 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.857050 4431 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:22.857143 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.857069 4431 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-console-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:22.857143 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.857079 4431 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5d73de5-3e17-4071-b707-2c12734ed196-oauth-serving-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:22.857143 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.857087 4431 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5d73de5-3e17-4071-b707-2c12734ed196-tmp\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:22.858625 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.858597 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-oauth-config" pod "f5d73de5-3e17-4071-b707-2c12734ed196" (UID: "f5d73de5-3e17-4071-b707-2c12734ed196"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:22.858723 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.858649 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d73de5-3e17-4071-b707-2c12734ed196-kube-api-access-s7m8l" pod "f5d73de5-3e17-4071-b707-2c12734ed196" (UID: "f5d73de5-3e17-4071-b707-2c12734ed196"). InnerVolumeSpecName "kube-api-access-s7m8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:22.858723 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.858684 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-serving-cert" pod "f5d73de5-3e17-4071-b707-2c12734ed196" (UID: "f5d73de5-3e17-4071-b707-2c12734ed196"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:22.957927 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.957902 4431 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-serving-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:22.957927 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.957923 4431 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5d73de5-3e17-4071-b707-2c12734ed196-console-oauth-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:22.958059 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:22.957933 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7m8l\" (UniqueName: \"kubernetes.io/projected/f5d73de5-3e17-4071-b707-2c12734ed196-kube-api-access-s7m8l\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:23.437624 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.437591 4431 generic.go:356] "Generic (PLEG): container finished" podID="f5d73de5-3e17-4071-b707-2c12734ed196" containerID="d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc" exitCode=0 Nov 02 13:58:23.437767 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.437645 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f8d7fffb-8lntm" Nov 02 13:58:23.437767 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.437674 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f8d7fffb-8lntm" event={"ID":"f5d73de5-3e17-4071-b707-2c12734ed196","Type":"ContainerDied","Data":"d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc"} Nov 02 13:58:23.437767 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.437745 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f8d7fffb-8lntm" event={"ID":"f5d73de5-3e17-4071-b707-2c12734ed196","Type":"ContainerDied","Data":"9f42158723c57de4170a4792b7b31ce22613dc371796bb53f55b840979d4a948"} Nov 02 13:58:23.437767 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.437766 4431 scope.go:122] "RemoveContainer" containerID="d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc" Nov 02 13:58:23.445648 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.445629 4431 scope.go:122] "RemoveContainer" containerID="d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc" Nov 02 13:58:23.445937 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:23.445917 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc\": container with ID starting with d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc not found: ID does not exist" containerID="d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc" Nov 02 13:58:23.446022 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.445942 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc"} err="failed to get container status \"d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc\": rpc error: code = NotFound desc = could not find container \"d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc\": container with ID starting with d515086d00e905abc03c77163015a32b91a8975fea6736d997de408793b2eccc not found: ID does not exist" Nov 02 13:58:23.492676 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.492655 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54f8d7fffb-8lntm"] Nov 02 13:58:23.506045 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:23.506022 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54f8d7fffb-8lntm"] Nov 02 13:58:25.073040 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:25.073011 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d73de5-3e17-4071-b707-2c12734ed196" path="/var/lib/kubelet/pods/f5d73de5-3e17-4071-b707-2c12734ed196/volumes" Nov 02 13:58:34.868597 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:34.868554 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:34.923615 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:34.921648 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:35.493735 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:35.493710 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:42.271334 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.271297 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:58:42.271980 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.271673 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="alertmanager" containerID="cri-o://3f28c2e7975088c0c85e6e9ab7acb15ff6e11d2e75aaaad961734d362c0c6602" gracePeriod=120 Nov 02 13:58:42.271980 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.271853 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy" containerID="cri-o://f743a84796ba585390512a199a30466a3952b620827f02944580b1ee1c51aa85" gracePeriod=120 Nov 02 13:58:42.271980 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.271946 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy-web" containerID="cri-o://b190ab69b25c5abea4b30ac71425ede143738a88dc5495711968e4489e0156ab" gracePeriod=120 Nov 02 13:58:42.271980 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.271941 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="prom-label-proxy" containerID="cri-o://ebb10bab3010feff012ec3b3c60486371e1ea3ab1fc6bc4a4eb0a63eb681fb81" gracePeriod=120 Nov 02 13:58:42.272150 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.272027 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="config-reloader" containerID="cri-o://f4d454c5cd40889173fd7b41d8e0e7f3dba1e87ae568c0d294a7db799414814f" gracePeriod=120 Nov 02 13:58:42.272150 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.272100 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy-metric" containerID="cri-o://353a7c61c845265ce519869588cc41db67bb81db679ccb6491c8d4b92737141d" gracePeriod=120 Nov 02 13:58:42.508908 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.508882 4431 generic.go:356] "Generic (PLEG): container finished" podID="18649210-246f-4218-8aa8-fb0f4c80095e" containerID="ebb10bab3010feff012ec3b3c60486371e1ea3ab1fc6bc4a4eb0a63eb681fb81" exitCode=0 Nov 02 13:58:42.508908 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.508905 4431 generic.go:356] "Generic (PLEG): container finished" podID="18649210-246f-4218-8aa8-fb0f4c80095e" containerID="353a7c61c845265ce519869588cc41db67bb81db679ccb6491c8d4b92737141d" exitCode=0 Nov 02 13:58:42.508908 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.508911 4431 generic.go:356] "Generic (PLEG): container finished" podID="18649210-246f-4218-8aa8-fb0f4c80095e" containerID="f743a84796ba585390512a199a30466a3952b620827f02944580b1ee1c51aa85" exitCode=0 Nov 02 13:58:42.509093 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.508917 4431 generic.go:356] "Generic (PLEG): container finished" podID="18649210-246f-4218-8aa8-fb0f4c80095e" containerID="f4d454c5cd40889173fd7b41d8e0e7f3dba1e87ae568c0d294a7db799414814f" exitCode=0 Nov 02 13:58:42.509093 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.508922 4431 generic.go:356] "Generic (PLEG): container finished" podID="18649210-246f-4218-8aa8-fb0f4c80095e" containerID="3f28c2e7975088c0c85e6e9ab7acb15ff6e11d2e75aaaad961734d362c0c6602" exitCode=0 Nov 02 13:58:42.509093 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.508959 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerDied","Data":"ebb10bab3010feff012ec3b3c60486371e1ea3ab1fc6bc4a4eb0a63eb681fb81"} Nov 02 13:58:42.509093 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.509036 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerDied","Data":"353a7c61c845265ce519869588cc41db67bb81db679ccb6491c8d4b92737141d"} Nov 02 13:58:42.509093 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.509065 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerDied","Data":"f743a84796ba585390512a199a30466a3952b620827f02944580b1ee1c51aa85"} Nov 02 13:58:42.509240 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.509095 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerDied","Data":"f4d454c5cd40889173fd7b41d8e0e7f3dba1e87ae568c0d294a7db799414814f"} Nov 02 13:58:42.509240 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:42.509119 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerDied","Data":"3f28c2e7975088c0c85e6e9ab7acb15ff6e11d2e75aaaad961734d362c0c6602"} Nov 02 13:58:43.516898 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.516867 4431 generic.go:356] "Generic (PLEG): container finished" podID="18649210-246f-4218-8aa8-fb0f4c80095e" containerID="b190ab69b25c5abea4b30ac71425ede143738a88dc5495711968e4489e0156ab" exitCode=0 Nov 02 13:58:43.517282 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.516935 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerDied","Data":"b190ab69b25c5abea4b30ac71425ede143738a88dc5495711968e4489e0156ab"} Nov 02 13:58:43.517282 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.517022 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18649210-246f-4218-8aa8-fb0f4c80095e","Type":"ContainerDied","Data":"f2b42363899f427f0e23367e99595ba3d962674e7878f42e9308f9f45f27b840"} Nov 02 13:58:43.517282 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.517065 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b42363899f427f0e23367e99595ba3d962674e7878f42e9308f9f45f27b840" Nov 02 13:58:43.518912 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.518893 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:43.614706 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614646 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-web\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614706 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614676 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-config-out\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614706 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614707 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-metrics-client-ca\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614967 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614732 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-cluster-tls-config\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614967 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614751 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-kube-api-access-pxd2g\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-kube-api-access-pxd2g\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614967 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614793 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-web-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-web-config\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614967 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614846 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-main-db\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614967 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614872 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-tls-assets\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-tls-assets\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614967 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614908 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.614967 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614939 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-main-tls\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.615331 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.614998 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.615331 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.615028 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-trusted-ca-bundle\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.615331 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.615062 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-config-volume\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-config-volume\") pod \"18649210-246f-4218-8aa8-fb0f4c80095e\" (UID: \"18649210-246f-4218-8aa8-fb0f4c80095e\") " Nov 02 13:58:43.615331 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.615104 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-metrics-client-ca" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:43.615534 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.615348 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-metrics-client-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.618896 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.617466 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-main-db" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:58:43.618896 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.617695 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-config-out" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:58:43.618896 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.618082 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-trusted-ca-bundle" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:43.618896 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.618123 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-metric" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:43.618896 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.618368 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-web" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:43.623691 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.623663 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-kube-api-access-pxd2g" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "kube-api-access-pxd2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:43.624855 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.624730 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-tls-assets" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:43.628446 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.628408 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-config-volume" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:43.629671 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.629632 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:43.629790 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.629693 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-main-tls" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:43.631756 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.631732 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-cluster-tls-config" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:43.636908 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.636881 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-web-config" pod "18649210-246f-4218-8aa8-fb0f4c80095e" (UID: "18649210-246f-4218-8aa8-fb0f4c80095e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:43.715721 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715693 4431 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-cluster-tls-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715721 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715716 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxd2g\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-kube-api-access-pxd2g\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715721 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715726 4431 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-web-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715735 4431 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-main-db\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715744 4431 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18649210-246f-4218-8aa8-fb0f4c80095e-tls-assets\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715753 4431 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715762 4431 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-main-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715771 4431 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715797 4431 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18649210-246f-4218-8aa8-fb0f4c80095e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715807 4431 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715818 4431 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18649210-246f-4218-8aa8-fb0f4c80095e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:43.715920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:43.715827 4431 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18649210-246f-4218-8aa8-fb0f4c80095e-config-out\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:44.520874 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.520846 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.599176 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.599121 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:58:44.612393 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.612369 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:58:44.636519 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636497 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:58:44.636801 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636770 4431 cpu_manager.go:414] "containerMap: removing container" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="init-config-reloader" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636802 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="init-config-reloader" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636824 4431 cpu_manager.go:414] "containerMap: removing container" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy-metric" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636829 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy-metric" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636841 4431 cpu_manager.go:414] "containerMap: removing container" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636846 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636854 4431 cpu_manager.go:414] "containerMap: removing container" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy-web" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636859 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy-web" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636870 4431 cpu_manager.go:414] "containerMap: removing container" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="config-reloader" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636877 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="config-reloader" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636885 4431 cpu_manager.go:414] "containerMap: removing container" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="alertmanager" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636889 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="alertmanager" Nov 02 13:58:44.636888 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636895 4431 cpu_manager.go:414] "containerMap: removing container" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="prom-label-proxy" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636899 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="prom-label-proxy" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636907 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f5d73de5-3e17-4071-b707-2c12734ed196" containerName="console" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636910 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d73de5-3e17-4071-b707-2c12734ed196" containerName="console" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636960 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy-metric" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636970 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636976 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="kube-rbac-proxy-web" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636982 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="config-reloader" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636987 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="alertmanager" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636991 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" containerName="prom-label-proxy" Nov 02 13:58:44.637486 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.636997 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d73de5-3e17-4071-b707-2c12734ed196" containerName="console" Nov 02 13:58:44.639895 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.639875 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.659342 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659318 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 13:58:44.659438 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659330 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 13:58:44.659438 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659315 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 13:58:44.659438 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659409 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 13:58:44.659638 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659622 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 13:58:44.659684 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659621 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 13:58:44.659684 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659651 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 13:58:44.659684 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659650 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 13:58:44.659768 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.659689 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bgfwj\"" Nov 02 13:58:44.673106 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.673087 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 13:58:44.674733 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.674709 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:58:44.724287 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724264 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22ae8630-0de4-45c5-86e7-ae97eef1e60b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724388 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724293 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-web-config\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724388 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724309 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724388 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724327 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22ae8630-0de4-45c5-86e7-ae97eef1e60b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724435 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22ae8630-0de4-45c5-86e7-ae97eef1e60b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724475 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22ae8630-0de4-45c5-86e7-ae97eef1e60b-config-out\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724572 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724495 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724604 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724578 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-config-volume\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724651 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724606 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22ae8630-0de4-45c5-86e7-ae97eef1e60b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724709 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724686 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvpkp\" (UniqueName: \"kubernetes.io/projected/22ae8630-0de4-45c5-86e7-ae97eef1e60b-kube-api-access-tvpkp\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724757 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724730 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724813 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724760 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.724844 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.724826 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826040 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826008 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22ae8630-0de4-45c5-86e7-ae97eef1e60b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826040 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826045 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvpkp\" (UniqueName: \"kubernetes.io/projected/22ae8630-0de4-45c5-86e7-ae97eef1e60b-kube-api-access-tvpkp\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826190 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826061 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826190 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826080 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826190 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826113 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826190 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826147 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22ae8630-0de4-45c5-86e7-ae97eef1e60b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826190 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826175 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-web-config\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826196 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826221 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22ae8630-0de4-45c5-86e7-ae97eef1e60b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826255 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22ae8630-0de4-45c5-86e7-ae97eef1e60b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826296 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22ae8630-0de4-45c5-86e7-ae97eef1e60b-config-out\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826322 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826415 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826364 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-config-volume\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.826707 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826664 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22ae8630-0de4-45c5-86e7-ae97eef1e60b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.827261 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.826990 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22ae8630-0de4-45c5-86e7-ae97eef1e60b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.829214 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829189 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.829214 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829204 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22ae8630-0de4-45c5-86e7-ae97eef1e60b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.829447 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829421 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.829447 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829438 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.829554 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829479 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22ae8630-0de4-45c5-86e7-ae97eef1e60b-config-out\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.829643 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829622 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-config-volume\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.829864 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829842 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.829943 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829867 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-web-config\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.830014 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.829998 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22ae8630-0de4-45c5-86e7-ae97eef1e60b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.831275 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.831256 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22ae8630-0de4-45c5-86e7-ae97eef1e60b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.850406 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.850357 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvpkp\" (UniqueName: \"kubernetes.io/projected/22ae8630-0de4-45c5-86e7-ae97eef1e60b-kube-api-access-tvpkp\") pod \"alertmanager-main-0\" (UID: \"22ae8630-0de4-45c5-86e7-ae97eef1e60b\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:44.956467 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:44.956447 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 13:58:45.072204 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:45.072174 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18649210-246f-4218-8aa8-fb0f4c80095e" path="/var/lib/kubelet/pods/18649210-246f-4218-8aa8-fb0f4c80095e/volumes" Nov 02 13:58:45.305401 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:45.305377 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 13:58:45.308417 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:58:45.308395 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ae8630_0de4_45c5_86e7_ae97eef1e60b.slice/crio-743082656ab3464806c002fda273b9cdd46bad312a958d87ee0e467970407dd7 WatchSource:0}: Error finding container 743082656ab3464806c002fda273b9cdd46bad312a958d87ee0e467970407dd7: Status 404 returned error can't find the container with id 743082656ab3464806c002fda273b9cdd46bad312a958d87ee0e467970407dd7 Nov 02 13:58:45.524948 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:45.524914 4431 generic.go:356] "Generic (PLEG): container finished" podID="22ae8630-0de4-45c5-86e7-ae97eef1e60b" containerID="1df2d0ed6577ffcaaaccda8b41a1e3cfb40cfbad3536ac82bb4224238d2234fd" exitCode=0 Nov 02 13:58:45.525267 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:45.524965 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22ae8630-0de4-45c5-86e7-ae97eef1e60b","Type":"ContainerDied","Data":"1df2d0ed6577ffcaaaccda8b41a1e3cfb40cfbad3536ac82bb4224238d2234fd"} Nov 02 13:58:45.525267 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:45.525025 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22ae8630-0de4-45c5-86e7-ae97eef1e60b","Type":"ContainerStarted","Data":"743082656ab3464806c002fda273b9cdd46bad312a958d87ee0e467970407dd7"} Nov 02 13:58:46.532278 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.532243 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22ae8630-0de4-45c5-86e7-ae97eef1e60b","Type":"ContainerStarted","Data":"e05bbf12d2d7c2d7cfee80291f81604b6fe490e085d9a01679d1e2f1b7a54fd5"} Nov 02 13:58:46.532765 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.532326 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22ae8630-0de4-45c5-86e7-ae97eef1e60b","Type":"ContainerStarted","Data":"56c1312a0de791d5a61dce7f45571c2ac7a2214f9396d68dfec247274406545a"} Nov 02 13:58:46.532765 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.532365 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22ae8630-0de4-45c5-86e7-ae97eef1e60b","Type":"ContainerStarted","Data":"af13877daff0d0c7324b964c90cd5318dc8f76bdd4edb354723afb57177ceae0"} Nov 02 13:58:46.532765 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.532405 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22ae8630-0de4-45c5-86e7-ae97eef1e60b","Type":"ContainerStarted","Data":"5e0066ed63891b208540657bf95b3512be99dc12e7ca82fd9f9c9faebca63039"} Nov 02 13:58:46.532765 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.532447 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22ae8630-0de4-45c5-86e7-ae97eef1e60b","Type":"ContainerStarted","Data":"e369b42fb9f47700ca000c7153d0024879b4c2313ab4d86692097e3ac57b2fdf"} Nov 02 13:58:46.532765 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.532484 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22ae8630-0de4-45c5-86e7-ae97eef1e60b","Type":"ContainerStarted","Data":"28a64628112518d15c388b1d789e52c130f92e88ff70830fa2fc910bb70e5f24"} Nov 02 13:58:46.607611 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.607547 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.5932181549999997 podStartE2EDuration="2.607531367s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.313212ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:46.603008649 +0000 UTC m=+91.640113580" watchObservedRunningTime="2026-11-02 13:58:46.607531367 +0000 UTC m=+91.644636299" Nov 02 13:58:46.970810 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.970756 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:58:46.971277 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.971135 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="prometheus" containerID="cri-o://56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" gracePeriod=600 Nov 02 13:58:46.971277 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.971166 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="thanos-sidecar" containerID="cri-o://701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" gracePeriod=600 Nov 02 13:58:46.971277 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.971156 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy" containerID="cri-o://374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" gracePeriod=600 Nov 02 13:58:46.971277 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.971192 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy-web" containerID="cri-o://8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" gracePeriod=600 Nov 02 13:58:46.971544 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.971190 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" gracePeriod=600 Nov 02 13:58:46.971544 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:46.971194 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="config-reloader" containerID="cri-o://62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" gracePeriod=600 Nov 02 13:58:47.210326 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.210299 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.249477 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249424 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-serving-certs-ca-bundle\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249477 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249458 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-trusted-ca-bundle\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249602 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249486 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249691 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249665 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249725 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249711 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-tls\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249791 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249737 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-metrics-client-certs\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249841 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249770 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-web-config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-web-config\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249841 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249817 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-kubelet-serving-ca-bundle\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249942 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249829 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-serving-certs-ca-bundle" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:47.249942 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249859 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-thanos-prometheus-http-client-file\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249942 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249888 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-0\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.249942 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249910 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-trusted-ca-bundle" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:47.250185 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249948 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-config-out\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250185 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.249979 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-kube-api-access-spvjk\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-kube-api-access-spvjk\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250185 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250011 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-1\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250185 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250058 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-kube-rbac-proxy\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250185 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250092 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-db\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250185 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250138 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-config\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250185 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250171 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-metrics-client-ca\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250546 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250205 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-2\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250546 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250244 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-grpc-tls\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250546 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250273 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-tls-assets\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-tls-assets\") pod \"07e7a2ac-2ac4-497e-942c-c1b44943876c\" (UID: \"07e7a2ac-2ac4-497e-942c-c1b44943876c\") " Nov 02 13:58:47.250546 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250301 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-kubelet-serving-ca-bundle" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:47.250546 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250528 4431 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.250546 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250548 4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.250867 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.250562 4431 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.251587 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.251260 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-2" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:47.251587 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.251314 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-metrics-client-ca" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:47.251861 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.251830 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-0" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:47.251984 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.251962 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-1" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:58:47.252139 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.252112 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-thanos-sidecar-tls" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.252734 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.252703 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-db" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:58:47.254196 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.254164 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.254290 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.254190 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-tls-assets" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:47.254290 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.254248 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-thanos-prometheus-http-client-file" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.254396 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.254376 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-config-out" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:58:47.254396 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.254377 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-metrics-client-certs" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.254496 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.254426 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-kube-api-access-spvjk" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "kube-api-access-spvjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:58:47.254565 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.254552 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-tls" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.254805 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.254787 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-kube-rbac-proxy" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.255105 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.255076 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-config" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.255105 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.255093 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-grpc-tls" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.262885 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.262864 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-web-config" pod "07e7a2ac-2ac4-497e-942c-c1b44943876c" (UID: "07e7a2ac-2ac4-497e-942c-c1b44943876c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:58:47.351290 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351262 4431 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351290 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351290 4431 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351306 4431 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351320 4431 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-metrics-client-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351335 4431 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-web-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351348 4431 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351361 4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351392 4431 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-config-out\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351405 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spvjk\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-kube-api-access-spvjk\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351418 4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351431 4431 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-kube-rbac-proxy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351444 4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-db\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351458 4431 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351472 4431 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-configmap-metrics-client-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351491 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351488 4431 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07e7a2ac-2ac4-497e-942c-c1b44943876c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351883 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351500 4431 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07e7a2ac-2ac4-497e-942c-c1b44943876c-secret-grpc-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.351883 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.351513 4431 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07e7a2ac-2ac4-497e-942c-c1b44943876c-tls-assets\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:58:47.539952 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.539881 4431 generic.go:356] "Generic (PLEG): container finished" podID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" exitCode=0 Nov 02 13:58:47.539952 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.539912 4431 generic.go:356] "Generic (PLEG): container finished" podID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" exitCode=0 Nov 02 13:58:47.539952 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.539919 4431 generic.go:356] "Generic (PLEG): container finished" podID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" exitCode=0 Nov 02 13:58:47.539952 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.539931 4431 generic.go:356] "Generic (PLEG): container finished" podID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" exitCode=0 Nov 02 13:58:47.539952 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.539938 4431 generic.go:356] "Generic (PLEG): container finished" podID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" exitCode=0 Nov 02 13:58:47.539952 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.539944 4431 generic.go:356] "Generic (PLEG): container finished" podID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" exitCode=0 Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.539962 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerDied","Data":"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0"} Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.540036 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerDied","Data":"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8"} Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.539973 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.540082 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerDied","Data":"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e"} Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.540123 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerDied","Data":"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900"} Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.540137 4431 scope.go:122] "RemoveContainer" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.540163 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerDied","Data":"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8"} Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.540202 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerDied","Data":"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444"} Nov 02 13:58:47.540621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.540242 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07e7a2ac-2ac4-497e-942c-c1b44943876c","Type":"ContainerDied","Data":"51dc8da85bd97520d1580db75784ef2bd7fffebbb092bc8a4092745fa36ddc8d"} Nov 02 13:58:47.548314 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.548173 4431 scope.go:122] "RemoveContainer" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" Nov 02 13:58:47.555049 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.555032 4431 scope.go:122] "RemoveContainer" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" Nov 02 13:58:47.561555 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.561536 4431 scope.go:122] "RemoveContainer" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" Nov 02 13:58:47.567621 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.567606 4431 scope.go:122] "RemoveContainer" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" Nov 02 13:58:47.573996 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.573982 4431 scope.go:122] "RemoveContainer" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" Nov 02 13:58:47.580694 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.580679 4431 scope.go:122] "RemoveContainer" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" Nov 02 13:58:47.588744 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.588725 4431 scope.go:122] "RemoveContainer" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" Nov 02 13:58:47.589017 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:47.588999 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": container with ID starting with 4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0 not found: ID does not exist" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" Nov 02 13:58:47.589079 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.589026 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0"} err="failed to get container status \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": rpc error: code = NotFound desc = could not find container \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": container with ID starting with 4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0 not found: ID does not exist" Nov 02 13:58:47.589079 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.589039 4431 scope.go:122] "RemoveContainer" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" Nov 02 13:58:47.589273 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:47.589255 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": container with ID starting with 374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8 not found: ID does not exist" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" Nov 02 13:58:47.589323 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.589277 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8"} err="failed to get container status \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": rpc error: code = NotFound desc = could not find container \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": container with ID starting with 374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8 not found: ID does not exist" Nov 02 13:58:47.589323 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.589291 4431 scope.go:122] "RemoveContainer" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" Nov 02 13:58:47.589517 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:47.589499 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": container with ID starting with 8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e not found: ID does not exist" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" Nov 02 13:58:47.589562 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.589524 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e"} err="failed to get container status \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": rpc error: code = NotFound desc = could not find container \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": container with ID starting with 8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e not found: ID does not exist" Nov 02 13:58:47.589562 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.589542 4431 scope.go:122] "RemoveContainer" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" Nov 02 13:58:47.589829 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:47.589813 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": container with ID starting with 701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900 not found: ID does not exist" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" Nov 02 13:58:47.589874 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.589833 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900"} err="failed to get container status \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": rpc error: code = NotFound desc = could not find container \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": container with ID starting with 701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900 not found: ID does not exist" Nov 02 13:58:47.589874 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.589847 4431 scope.go:122] "RemoveContainer" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" Nov 02 13:58:47.590072 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:47.590057 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": container with ID starting with 62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8 not found: ID does not exist" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" Nov 02 13:58:47.590110 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.590076 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8"} err="failed to get container status \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": rpc error: code = NotFound desc = could not find container \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": container with ID starting with 62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8 not found: ID does not exist" Nov 02 13:58:47.590110 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.590089 4431 scope.go:122] "RemoveContainer" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" Nov 02 13:58:47.590351 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:47.590335 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": container with ID starting with 56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444 not found: ID does not exist" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" Nov 02 13:58:47.590398 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.590355 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444"} err="failed to get container status \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": rpc error: code = NotFound desc = could not find container \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": container with ID starting with 56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444 not found: ID does not exist" Nov 02 13:58:47.590398 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.590368 4431 scope.go:122] "RemoveContainer" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" Nov 02 13:58:47.590679 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:47.590664 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": container with ID starting with b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6 not found: ID does not exist" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" Nov 02 13:58:47.590722 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.590682 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6"} err="failed to get container status \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": rpc error: code = NotFound desc = could not find container \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": container with ID starting with b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6 not found: ID does not exist" Nov 02 13:58:47.590722 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.590697 4431 scope.go:122] "RemoveContainer" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" Nov 02 13:58:47.591046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591031 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0"} err="failed to get container status \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": rpc error: code = NotFound desc = could not find container \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": container with ID starting with 4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0 not found: ID does not exist" Nov 02 13:58:47.591046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591045 4431 scope.go:122] "RemoveContainer" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" Nov 02 13:58:47.591251 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591234 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8"} err="failed to get container status \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": rpc error: code = NotFound desc = could not find container \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": container with ID starting with 374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8 not found: ID does not exist" Nov 02 13:58:47.591292 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591252 4431 scope.go:122] "RemoveContainer" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" Nov 02 13:58:47.591448 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591432 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e"} err="failed to get container status \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": rpc error: code = NotFound desc = could not find container \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": container with ID starting with 8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e not found: ID does not exist" Nov 02 13:58:47.591494 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591449 4431 scope.go:122] "RemoveContainer" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" Nov 02 13:58:47.591636 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591620 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900"} err="failed to get container status \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": rpc error: code = NotFound desc = could not find container \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": container with ID starting with 701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900 not found: ID does not exist" Nov 02 13:58:47.591685 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591636 4431 scope.go:122] "RemoveContainer" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" Nov 02 13:58:47.591827 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591812 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8"} err="failed to get container status \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": rpc error: code = NotFound desc = could not find container \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": container with ID starting with 62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8 not found: ID does not exist" Nov 02 13:58:47.591875 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591828 4431 scope.go:122] "RemoveContainer" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" Nov 02 13:58:47.592003 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.591988 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444"} err="failed to get container status \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": rpc error: code = NotFound desc = could not find container \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": container with ID starting with 56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444 not found: ID does not exist" Nov 02 13:58:47.592047 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592003 4431 scope.go:122] "RemoveContainer" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" Nov 02 13:58:47.592179 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592162 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6"} err="failed to get container status \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": rpc error: code = NotFound desc = could not find container \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": container with ID starting with b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6 not found: ID does not exist" Nov 02 13:58:47.592222 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592178 4431 scope.go:122] "RemoveContainer" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" Nov 02 13:58:47.592347 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592332 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0"} err="failed to get container status \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": rpc error: code = NotFound desc = could not find container \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": container with ID starting with 4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0 not found: ID does not exist" Nov 02 13:58:47.592388 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592347 4431 scope.go:122] "RemoveContainer" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" Nov 02 13:58:47.592515 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592500 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8"} err="failed to get container status \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": rpc error: code = NotFound desc = could not find container \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": container with ID starting with 374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8 not found: ID does not exist" Nov 02 13:58:47.592558 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592516 4431 scope.go:122] "RemoveContainer" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" Nov 02 13:58:47.592720 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592704 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e"} err="failed to get container status \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": rpc error: code = NotFound desc = could not find container \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": container with ID starting with 8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e not found: ID does not exist" Nov 02 13:58:47.592760 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592719 4431 scope.go:122] "RemoveContainer" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" Nov 02 13:58:47.592928 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592912 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900"} err="failed to get container status \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": rpc error: code = NotFound desc = could not find container \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": container with ID starting with 701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900 not found: ID does not exist" Nov 02 13:58:47.592972 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.592928 4431 scope.go:122] "RemoveContainer" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" Nov 02 13:58:47.593112 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593096 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8"} err="failed to get container status \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": rpc error: code = NotFound desc = could not find container \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": container with ID starting with 62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8 not found: ID does not exist" Nov 02 13:58:47.593152 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593111 4431 scope.go:122] "RemoveContainer" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" Nov 02 13:58:47.593290 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593275 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444"} err="failed to get container status \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": rpc error: code = NotFound desc = could not find container \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": container with ID starting with 56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444 not found: ID does not exist" Nov 02 13:58:47.593327 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593290 4431 scope.go:122] "RemoveContainer" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" Nov 02 13:58:47.593469 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593454 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6"} err="failed to get container status \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": rpc error: code = NotFound desc = could not find container \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": container with ID starting with b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6 not found: ID does not exist" Nov 02 13:58:47.593509 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593469 4431 scope.go:122] "RemoveContainer" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" Nov 02 13:58:47.593653 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593637 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0"} err="failed to get container status \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": rpc error: code = NotFound desc = could not find container \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": container with ID starting with 4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0 not found: ID does not exist" Nov 02 13:58:47.593732 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593653 4431 scope.go:122] "RemoveContainer" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" Nov 02 13:58:47.593842 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593826 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8"} err="failed to get container status \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": rpc error: code = NotFound desc = could not find container \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": container with ID starting with 374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8 not found: ID does not exist" Nov 02 13:58:47.593889 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.593842 4431 scope.go:122] "RemoveContainer" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" Nov 02 13:58:47.594015 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594000 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e"} err="failed to get container status \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": rpc error: code = NotFound desc = could not find container \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": container with ID starting with 8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e not found: ID does not exist" Nov 02 13:58:47.594015 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594014 4431 scope.go:122] "RemoveContainer" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" Nov 02 13:58:47.594188 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594173 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900"} err="failed to get container status \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": rpc error: code = NotFound desc = could not find container \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": container with ID starting with 701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900 not found: ID does not exist" Nov 02 13:58:47.594228 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594187 4431 scope.go:122] "RemoveContainer" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" Nov 02 13:58:47.594357 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594341 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8"} err="failed to get container status \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": rpc error: code = NotFound desc = could not find container \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": container with ID starting with 62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8 not found: ID does not exist" Nov 02 13:58:47.594396 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594356 4431 scope.go:122] "RemoveContainer" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" Nov 02 13:58:47.594544 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594529 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444"} err="failed to get container status \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": rpc error: code = NotFound desc = could not find container \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": container with ID starting with 56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444 not found: ID does not exist" Nov 02 13:58:47.594584 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594544 4431 scope.go:122] "RemoveContainer" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" Nov 02 13:58:47.594711 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594696 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6"} err="failed to get container status \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": rpc error: code = NotFound desc = could not find container \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": container with ID starting with b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6 not found: ID does not exist" Nov 02 13:58:47.594761 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594711 4431 scope.go:122] "RemoveContainer" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" Nov 02 13:58:47.594895 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594879 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0"} err="failed to get container status \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": rpc error: code = NotFound desc = could not find container \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": container with ID starting with 4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0 not found: ID does not exist" Nov 02 13:58:47.594935 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.594894 4431 scope.go:122] "RemoveContainer" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" Nov 02 13:58:47.595143 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.595116 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8"} err="failed to get container status \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": rpc error: code = NotFound desc = could not find container \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": container with ID starting with 374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8 not found: ID does not exist" Nov 02 13:58:47.595143 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.595142 4431 scope.go:122] "RemoveContainer" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" Nov 02 13:58:47.595377 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.595357 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e"} err="failed to get container status \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": rpc error: code = NotFound desc = could not find container \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": container with ID starting with 8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e not found: ID does not exist" Nov 02 13:58:47.595450 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.595377 4431 scope.go:122] "RemoveContainer" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" Nov 02 13:58:47.596526 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.596497 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900"} err="failed to get container status \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": rpc error: code = NotFound desc = could not find container \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": container with ID starting with 701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900 not found: ID does not exist" Nov 02 13:58:47.596526 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.596525 4431 scope.go:122] "RemoveContainer" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" Nov 02 13:58:47.596912 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.596812 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8"} err="failed to get container status \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": rpc error: code = NotFound desc = could not find container \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": container with ID starting with 62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8 not found: ID does not exist" Nov 02 13:58:47.596912 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.596834 4431 scope.go:122] "RemoveContainer" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" Nov 02 13:58:47.597126 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.597107 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444"} err="failed to get container status \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": rpc error: code = NotFound desc = could not find container \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": container with ID starting with 56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444 not found: ID does not exist" Nov 02 13:58:47.597172 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.597127 4431 scope.go:122] "RemoveContainer" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" Nov 02 13:58:47.597398 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.597376 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6"} err="failed to get container status \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": rpc error: code = NotFound desc = could not find container \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": container with ID starting with b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6 not found: ID does not exist" Nov 02 13:58:47.597398 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.597396 4431 scope.go:122] "RemoveContainer" containerID="4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0" Nov 02 13:58:47.597717 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.597693 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0"} err="failed to get container status \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": rpc error: code = NotFound desc = could not find container \"4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0\": container with ID starting with 4c8026ff02323239a9c073d56ba8404d19238e2312526c59b39dbf73cdbb28f0 not found: ID does not exist" Nov 02 13:58:47.597717 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.597715 4431 scope.go:122] "RemoveContainer" containerID="374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8" Nov 02 13:58:47.598008 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.597990 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8"} err="failed to get container status \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": rpc error: code = NotFound desc = could not find container \"374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8\": container with ID starting with 374d9cd2ec5c7c12953db843aa2bf058c978d14ccd8b041d7f5736e65b406ed8 not found: ID does not exist" Nov 02 13:58:47.598057 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598009 4431 scope.go:122] "RemoveContainer" containerID="8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e" Nov 02 13:58:47.598217 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598199 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e"} err="failed to get container status \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": rpc error: code = NotFound desc = could not find container \"8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e\": container with ID starting with 8712f506f6c1c98c1a6fdea98029b33bcace0c66695592217d342b5204836b1e not found: ID does not exist" Nov 02 13:58:47.598287 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598219 4431 scope.go:122] "RemoveContainer" containerID="701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900" Nov 02 13:58:47.598404 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598389 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:58:47.598463 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598447 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900"} err="failed to get container status \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": rpc error: code = NotFound desc = could not find container \"701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900\": container with ID starting with 701e1f15b2d99667c615ee1f792716ff8a761cf51b6eca8009dd9bebdd46e900 not found: ID does not exist" Nov 02 13:58:47.598501 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598464 4431 scope.go:122] "RemoveContainer" containerID="62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8" Nov 02 13:58:47.598675 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598658 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8"} err="failed to get container status \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": rpc error: code = NotFound desc = could not find container \"62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8\": container with ID starting with 62e3a0633c8137a32746ca95eae3506d20091525daefcfd3f2b1cad440ec40e8 not found: ID does not exist" Nov 02 13:58:47.598717 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598675 4431 scope.go:122] "RemoveContainer" containerID="56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444" Nov 02 13:58:47.598898 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598881 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444"} err="failed to get container status \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": rpc error: code = NotFound desc = could not find container \"56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444\": container with ID starting with 56b66109a1bc9b1b6a2e119a39777e8134a9b628b142fc019a7c4e68a332e444 not found: ID does not exist" Nov 02 13:58:47.598947 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.598898 4431 scope.go:122] "RemoveContainer" containerID="b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6" Nov 02 13:58:47.599089 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.599073 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6"} err="failed to get container status \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": rpc error: code = NotFound desc = could not find container \"b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6\": container with ID starting with b95dc0b06af51e8e2f6bdce230eae325f0ce256115d1a3519f10cefd39c14cc6 not found: ID does not exist" Nov 02 13:58:47.611378 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.611356 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:58:47.641304 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.641280 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:58:47.641929 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.641911 4431 cpu_manager.go:414] "containerMap: removing container" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="init-config-reloader" Nov 02 13:58:47.642038 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.641931 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="init-config-reloader" Nov 02 13:58:47.642038 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.641962 4431 cpu_manager.go:414] "containerMap: removing container" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="config-reloader" Nov 02 13:58:47.642038 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.641971 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="config-reloader" Nov 02 13:58:47.642038 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.641991 4431 cpu_manager.go:414] "containerMap: removing container" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy-thanos" Nov 02 13:58:47.642038 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.641998 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy-thanos" Nov 02 13:58:47.642038 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642012 4431 cpu_manager.go:414] "containerMap: removing container" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy-web" Nov 02 13:58:47.642038 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642025 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy-web" Nov 02 13:58:47.642038 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642041 4431 cpu_manager.go:414] "containerMap: removing container" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="prometheus" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642048 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="prometheus" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642111 4431 cpu_manager.go:414] "containerMap: removing container" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642119 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642135 4431 cpu_manager.go:414] "containerMap: removing container" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="thanos-sidecar" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642141 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="thanos-sidecar" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642250 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642265 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="thanos-sidecar" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642276 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="config-reloader" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642293 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy-thanos" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642302 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="kube-rbac-proxy-web" Nov 02 13:58:47.642495 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.642318 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" containerName="prometheus" Nov 02 13:58:47.647662 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.647645 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.669474 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.669431 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 13:58:47.669895 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.669867 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 13:58:47.669895 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.669871 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 13:58:47.670014 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.669902 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 13:58:47.670014 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.669915 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 13:58:47.670166 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670152 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 13:58:47.670166 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670160 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6l2d5\"" Nov 02 13:58:47.670240 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670162 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 13:58:47.670273 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670262 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 13:58:47.670333 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670320 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 13:58:47.670379 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670324 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 13:58:47.670466 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670454 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 13:58:47.670659 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670642 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 13:58:47.670704 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670660 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 13:58:47.670751 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.670724 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-inmegb4reb5t\"" Nov 02 13:58:47.684210 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.684188 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 13:58:47.684793 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.684745 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 13:58:47.686099 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.686078 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:58:47.754838 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.754814 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.754923 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.754847 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.754923 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.754865 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.754923 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.754886 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755095 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.754952 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755095 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.754995 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755095 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755075 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755243 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755117 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755243 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755145 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755243 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755180 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac83895e-0242-4235-970e-eb73c99fa77f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755243 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755210 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-config\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755426 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755247 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755426 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755297 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755426 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755371 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fvr\" (UniqueName: \"kubernetes.io/projected/ac83895e-0242-4235-970e-eb73c99fa77f-kube-api-access-l7fvr\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755426 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755408 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755594 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755455 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac83895e-0242-4235-970e-eb73c99fa77f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755594 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755473 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755594 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755488 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755594 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755519 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.755594 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.755553 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.856839 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.856746 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.856839 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.856817 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.856984 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.856846 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac83895e-0242-4235-970e-eb73c99fa77f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.856984 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.856874 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-config\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.856984 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.856897 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.856984 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.856923 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.856984 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.856952 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fvr\" (UniqueName: \"kubernetes.io/projected/ac83895e-0242-4235-970e-eb73c99fa77f-kube-api-access-l7fvr\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857235 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.856986 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857235 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857018 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac83895e-0242-4235-970e-eb73c99fa77f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857235 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857046 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857235 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857071 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857235 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857102 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857235 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857136 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857235 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857185 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857235 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857229 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857638 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857254 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857638 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857292 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857638 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857320 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857638 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857349 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857638 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857379 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857638 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857382 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.857638 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857456 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.858009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857823 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.858009 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.857848 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.858340 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.858147 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.859769 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.859711 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac83895e-0242-4235-970e-eb73c99fa77f-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.860544 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.860389 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.860544 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.860445 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.860544 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.860516 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.861510 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.860639 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac83895e-0242-4235-970e-eb73c99fa77f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.861510 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.860755 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.861510 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.861008 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.861510 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.861193 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.861510 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.861308 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.861510 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.861449 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-config\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.862422 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.862400 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.862859 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.862838 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.863111 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.863090 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac83895e-0242-4235-970e-eb73c99fa77f-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.863918 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.863901 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac83895e-0242-4235-970e-eb73c99fa77f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.881833 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.881812 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fvr\" (UniqueName: \"kubernetes.io/projected/ac83895e-0242-4235-970e-eb73c99fa77f-kube-api-access-l7fvr\") pod \"prometheus-k8s-0\" (UID: \"ac83895e-0242-4235-970e-eb73c99fa77f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:47.963920 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:47.963890 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:58:48.304554 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:48.304532 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 13:58:48.304759 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:58:48.304739 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac83895e_0242_4235_970e_eb73c99fa77f.slice/crio-32a983f1684f8d93e79f4e35359b8c44f1042f3014559dd1cbf4780435ae60e0 WatchSource:0}: Error finding container 32a983f1684f8d93e79f4e35359b8c44f1042f3014559dd1cbf4780435ae60e0: Status 404 returned error can't find the container with id 32a983f1684f8d93e79f4e35359b8c44f1042f3014559dd1cbf4780435ae60e0 Nov 02 13:58:48.545990 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:48.545963 4431 generic.go:356] "Generic (PLEG): container finished" podID="ac83895e-0242-4235-970e-eb73c99fa77f" containerID="7c38c4dcb0c0ecc754ee6f5151d26cd9dbb471b8d3b877255dd3bbe82a659c03" exitCode=0 Nov 02 13:58:48.546356 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:48.546056 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac83895e-0242-4235-970e-eb73c99fa77f","Type":"ContainerDied","Data":"7c38c4dcb0c0ecc754ee6f5151d26cd9dbb471b8d3b877255dd3bbe82a659c03"} Nov 02 13:58:48.546356 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:48.546129 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac83895e-0242-4235-970e-eb73c99fa77f","Type":"ContainerStarted","Data":"32a983f1684f8d93e79f4e35359b8c44f1042f3014559dd1cbf4780435ae60e0"} Nov 02 13:58:49.074078 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:49.074007 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e7a2ac-2ac4-497e-942c-c1b44943876c" path="/var/lib/kubelet/pods/07e7a2ac-2ac4-497e-942c-c1b44943876c/volumes" Nov 02 13:58:49.554626 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:49.554592 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac83895e-0242-4235-970e-eb73c99fa77f","Type":"ContainerStarted","Data":"7ed536a294c06edf963f82778ac82a48626f49e13d5679d81e00c370021cb856"} Nov 02 13:58:49.555019 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:49.554658 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac83895e-0242-4235-970e-eb73c99fa77f","Type":"ContainerStarted","Data":"f001e24f3ed35c2c8ad414e8415ef88f72ae79259950f619b1fdb309f21e0047"} Nov 02 13:58:49.555019 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:49.554685 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac83895e-0242-4235-970e-eb73c99fa77f","Type":"ContainerStarted","Data":"2092f7b1eeee355927f28a2ec03ff02d5a34f0a2cdbcead08998cecd1d15288a"} Nov 02 13:58:49.555019 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:49.554709 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac83895e-0242-4235-970e-eb73c99fa77f","Type":"ContainerStarted","Data":"e9ac87f1896addd56b74445a95552eb472c3ef4f8c8a0ccfa81115a8722ce601"} Nov 02 13:58:49.555019 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:49.554734 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac83895e-0242-4235-970e-eb73c99fa77f","Type":"ContainerStarted","Data":"64b1a02734b4259fc5a535aae5135c43c59fb0c95e26003a2371390d4124c97e"} Nov 02 13:58:49.555019 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:49.554759 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac83895e-0242-4235-970e-eb73c99fa77f","Type":"ContainerStarted","Data":"7f5e74057b6579a2f757a8dd3e4372cd4a52ef7bb95bad92cfdf9c13d9e9a8a0"} Nov 02 13:58:49.635010 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:49.634964 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.620775934 podStartE2EDuration="2.634949064s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.17313ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:49.634337411 +0000 UTC m=+94.671442343" watchObservedRunningTime="2026-11-02 13:58:49.634949064 +0000 UTC m=+94.672054031" Nov 02 13:58:50.213663 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.213634 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74468c9945-7qj7c"] Nov 02 13:58:50.216323 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.216304 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.242125 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.242052 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74468c9945-7qj7c"] Nov 02 13:58:50.277116 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.277093 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-service-ca\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.277250 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.277121 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-trusted-ca-bundle\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.277250 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.277142 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-serving-cert\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.277250 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.277226 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-config\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.277350 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.277278 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-oauth-config\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.277350 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.277305 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-oauth-serving-cert\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.277456 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.277440 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdwmt\" (UniqueName: \"kubernetes.io/projected/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-kube-api-access-qdwmt\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.277508 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.277495 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-tmp\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.378430 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.378391 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwmt\" (UniqueName: \"kubernetes.io/projected/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-kube-api-access-qdwmt\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.378590 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.378461 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-tmp\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.378590 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.378503 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-service-ca\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.378590 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.378528 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-trusted-ca-bundle\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.378590 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.378557 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-serving-cert\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.378832 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.378593 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-config\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.378935 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.378911 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-tmp\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.379422 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.379369 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-config\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.379543 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.379479 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-trusted-ca-bundle\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.379624 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.378624 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-oauth-config\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.379744 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.379729 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-oauth-serving-cert\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.380095 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.380072 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-service-ca\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.380682 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.380662 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-oauth-serving-cert\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.381205 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.381180 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-oauth-config\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.381661 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.381640 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-serving-cert\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.408453 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.408403 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwmt\" (UniqueName: \"kubernetes.io/projected/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-kube-api-access-qdwmt\") pod \"console-74468c9945-7qj7c\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.527759 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.527687 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:58:50.862852 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:50.860966 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74468c9945-7qj7c"] Nov 02 13:58:50.872565 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:58:50.872536 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa9d0eb_6097_4ddf_968a_2e085fb2a45f.slice/crio-f5dd1d6a3fe3ff9e1d5905fa88994697cc535bab5d88c032cc53e243c8958076 WatchSource:0}: Error finding container f5dd1d6a3fe3ff9e1d5905fa88994697cc535bab5d88c032cc53e243c8958076: Status 404 returned error can't find the container with id f5dd1d6a3fe3ff9e1d5905fa88994697cc535bab5d88c032cc53e243c8958076 Nov 02 13:58:51.291293 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:58:51.291253 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 13:58:51.565456 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:51.565366 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74468c9945-7qj7c" event={"ID":"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f","Type":"ContainerStarted","Data":"5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971"} Nov 02 13:58:51.565456 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:51.565427 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74468c9945-7qj7c" event={"ID":"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f","Type":"ContainerStarted","Data":"f5dd1d6a3fe3ff9e1d5905fa88994697cc535bab5d88c032cc53e243c8958076"} Nov 02 13:58:51.615982 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:51.615923 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74468c9945-7qj7c" podStartSLOduration=1.615910746 podStartE2EDuration="1.615910746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:51.615190037 +0000 UTC m=+96.652294970" watchObservedRunningTime="2026-11-02 13:58:51.615910746 +0000 UTC m=+96.653015726" Nov 02 13:58:52.964366 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:58:52.964336 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:59:00.528219 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:00.528172 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:59:00.528632 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:00.528250 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:59:00.532789 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:00.532756 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:59:00.605468 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:00.605438 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74468c9945-7qj7c" Nov 02 13:59:00.698063 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:00.698037 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574cbdb96f-cg9k7"] Nov 02 13:59:21.283472 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:59:21.283430 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 13:59:25.717902 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:25.717843 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-574cbdb96f-cg9k7" podUID="ef25160b-7ff9-42e3-9ba2-af126544bac3" containerName="console" containerID="cri-o://f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3" gracePeriod=15 Nov 02 13:59:26.001818 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.001727 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:59:26.168201 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168172 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-service-ca\") pod \"ef25160b-7ff9-42e3-9ba2-af126544bac3\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " Nov 02 13:59:26.168360 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168226 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef25160b-7ff9-42e3-9ba2-af126544bac3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef25160b-7ff9-42e3-9ba2-af126544bac3-tmp\") pod \"ef25160b-7ff9-42e3-9ba2-af126544bac3\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " Nov 02 13:59:26.168360 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168259 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-config\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-config\") pod \"ef25160b-7ff9-42e3-9ba2-af126544bac3\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " Nov 02 13:59:26.168360 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168296 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-oauth-serving-cert\") pod \"ef25160b-7ff9-42e3-9ba2-af126544bac3\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " Nov 02 13:59:26.168360 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168320 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-trusted-ca-bundle\") pod \"ef25160b-7ff9-42e3-9ba2-af126544bac3\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " Nov 02 13:59:26.168360 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168348 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef25160b-7ff9-42e3-9ba2-af126544bac3-kube-api-access-7szrt\" (UniqueName: \"kubernetes.io/projected/ef25160b-7ff9-42e3-9ba2-af126544bac3-kube-api-access-7szrt\") pod \"ef25160b-7ff9-42e3-9ba2-af126544bac3\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " Nov 02 13:59:26.168604 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168375 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-serving-cert\") pod \"ef25160b-7ff9-42e3-9ba2-af126544bac3\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " Nov 02 13:59:26.168604 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168425 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-oauth-config\") pod \"ef25160b-7ff9-42e3-9ba2-af126544bac3\" (UID: \"ef25160b-7ff9-42e3-9ba2-af126544bac3\") " Nov 02 13:59:26.168604 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168573 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef25160b-7ff9-42e3-9ba2-af126544bac3-tmp" pod "ef25160b-7ff9-42e3-9ba2-af126544bac3" (UID: "ef25160b-7ff9-42e3-9ba2-af126544bac3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:59:26.168745 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168665 4431 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef25160b-7ff9-42e3-9ba2-af126544bac3-tmp\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:59:26.168745 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168713 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-service-ca" pod "ef25160b-7ff9-42e3-9ba2-af126544bac3" (UID: "ef25160b-7ff9-42e3-9ba2-af126544bac3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:59:26.168883 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168814 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-oauth-serving-cert" pod "ef25160b-7ff9-42e3-9ba2-af126544bac3" (UID: "ef25160b-7ff9-42e3-9ba2-af126544bac3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:59:26.168883 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168826 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-config" pod "ef25160b-7ff9-42e3-9ba2-af126544bac3" (UID: "ef25160b-7ff9-42e3-9ba2-af126544bac3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:59:26.168981 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.168885 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-trusted-ca-bundle" pod "ef25160b-7ff9-42e3-9ba2-af126544bac3" (UID: "ef25160b-7ff9-42e3-9ba2-af126544bac3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:59:26.170653 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.170622 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-oauth-config" pod "ef25160b-7ff9-42e3-9ba2-af126544bac3" (UID: "ef25160b-7ff9-42e3-9ba2-af126544bac3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:59:26.170752 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.170666 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-serving-cert" pod "ef25160b-7ff9-42e3-9ba2-af126544bac3" (UID: "ef25160b-7ff9-42e3-9ba2-af126544bac3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:59:26.170752 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.170679 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef25160b-7ff9-42e3-9ba2-af126544bac3-kube-api-access-7szrt" pod "ef25160b-7ff9-42e3-9ba2-af126544bac3" (UID: "ef25160b-7ff9-42e3-9ba2-af126544bac3"). InnerVolumeSpecName "kube-api-access-7szrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:59:26.269279 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.269231 4431 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:59:26.269279 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.269251 4431 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:59:26.269279 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.269260 4431 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-oauth-serving-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:59:26.269279 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.269269 4431 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef25160b-7ff9-42e3-9ba2-af126544bac3-trusted-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:59:26.269279 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.269277 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szrt\" (UniqueName: \"kubernetes.io/projected/ef25160b-7ff9-42e3-9ba2-af126544bac3-kube-api-access-7szrt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:59:26.269481 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.269286 4431 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-serving-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:59:26.269481 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.269294 4431 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef25160b-7ff9-42e3-9ba2-af126544bac3-console-oauth-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 13:59:26.700073 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.700041 4431 generic.go:356] "Generic (PLEG): container finished" podID="ef25160b-7ff9-42e3-9ba2-af126544bac3" containerID="f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3" exitCode=0 Nov 02 13:59:26.700231 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.700093 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574cbdb96f-cg9k7" Nov 02 13:59:26.700231 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.700125 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574cbdb96f-cg9k7" event={"ID":"ef25160b-7ff9-42e3-9ba2-af126544bac3","Type":"ContainerDied","Data":"f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3"} Nov 02 13:59:26.700231 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.700177 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574cbdb96f-cg9k7" event={"ID":"ef25160b-7ff9-42e3-9ba2-af126544bac3","Type":"ContainerDied","Data":"34131a06f159786aaf1f5273042bf10414e1d5b7644706d05575aaa61c207fca"} Nov 02 13:59:26.700231 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.700203 4431 scope.go:122] "RemoveContainer" containerID="f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3" Nov 02 13:59:26.709203 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.709188 4431 scope.go:122] "RemoveContainer" containerID="f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3" Nov 02 13:59:26.709454 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:59:26.709437 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3\": container with ID starting with f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3 not found: ID does not exist" containerID="f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3" Nov 02 13:59:26.709493 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.709464 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3"} err="failed to get container status \"f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3\": rpc error: code = NotFound desc = could not find container \"f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3\": container with ID starting with f700fc0a24064e0076f1a37a555e337c5fc3baafb97cc7714468d45e1f8a5ba3 not found: ID does not exist" Nov 02 13:59:26.757427 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.757400 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574cbdb96f-cg9k7"] Nov 02 13:59:26.769928 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:26.769907 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-574cbdb96f-cg9k7"] Nov 02 13:59:27.077049 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:27.076977 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef25160b-7ff9-42e3-9ba2-af126544bac3" path="/var/lib/kubelet/pods/ef25160b-7ff9-42e3-9ba2-af126544bac3/volumes" Nov 02 13:59:47.964636 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:47.964544 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:59:47.981181 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:47.981153 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:59:48.806614 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:48.806573 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 13:59:51.281507 ip-10-0-134-9 kubenswrapper[4431]: E1102 13:59:51.281466 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 13:59:58.524912 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.524874 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-687865856c-95sjw"] Nov 02 13:59:58.525401 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.525310 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ef25160b-7ff9-42e3-9ba2-af126544bac3" containerName="console" Nov 02 13:59:58.525401 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.525330 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef25160b-7ff9-42e3-9ba2-af126544bac3" containerName="console" Nov 02 13:59:58.525519 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.525448 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef25160b-7ff9-42e3-9ba2-af126544bac3" containerName="console" Nov 02 13:59:58.528259 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.528236 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.551872 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.551846 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-687865856c-95sjw"] Nov 02 13:59:58.587174 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.587146 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-service-ca\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.587284 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.587181 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-config\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.587284 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.587198 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-oauth-serving-cert\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.587284 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.587267 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz755\" (UniqueName: \"kubernetes.io/projected/605e26d1-e6bb-4f8e-8b33-3c9277139663-kube-api-access-pz755\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.587412 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.587320 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/605e26d1-e6bb-4f8e-8b33-3c9277139663-tmp\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.587446 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.587425 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-serving-cert\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.587479 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.587444 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-trusted-ca-bundle\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.587479 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.587465 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-oauth-config\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688239 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688211 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-oauth-serving-cert\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688352 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688257 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz755\" (UniqueName: \"kubernetes.io/projected/605e26d1-e6bb-4f8e-8b33-3c9277139663-kube-api-access-pz755\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688352 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688281 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/605e26d1-e6bb-4f8e-8b33-3c9277139663-tmp\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688352 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688314 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-serving-cert\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688352 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688338 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-trusted-ca-bundle\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688565 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688480 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-oauth-config\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688565 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688546 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-service-ca\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688668 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688586 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-config\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.688830 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.688800 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/605e26d1-e6bb-4f8e-8b33-3c9277139663-tmp\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.689068 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.689042 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-oauth-serving-cert\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.689227 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.689132 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-trusted-ca-bundle\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.689302 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.689242 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-config\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.689302 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.689271 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/605e26d1-e6bb-4f8e-8b33-3c9277139663-service-ca\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.690739 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.690718 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-serving-cert\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.690838 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.690769 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/605e26d1-e6bb-4f8e-8b33-3c9277139663-console-oauth-config\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.713046 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.713016 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz755\" (UniqueName: \"kubernetes.io/projected/605e26d1-e6bb-4f8e-8b33-3c9277139663-kube-api-access-pz755\") pod \"console-687865856c-95sjw\" (UID: \"605e26d1-e6bb-4f8e-8b33-3c9277139663\") " pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:58.840804 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:58.840749 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-687865856c-95sjw" Nov 02 13:59:59.143721 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:59.143693 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-687865856c-95sjw"] Nov 02 13:59:59.145503 ip-10-0-134-9 kubenswrapper[4431]: W1102 13:59:59.145478 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605e26d1_e6bb_4f8e_8b33_3c9277139663.slice/crio-6027abd04663af8f4e6d9b613f4971d71dc60fe1b848d2710d534551e1228cfc WatchSource:0}: Error finding container 6027abd04663af8f4e6d9b613f4971d71dc60fe1b848d2710d534551e1228cfc: Status 404 returned error can't find the container with id 6027abd04663af8f4e6d9b613f4971d71dc60fe1b848d2710d534551e1228cfc Nov 02 13:59:59.827196 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:59.827157 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-687865856c-95sjw" event={"ID":"605e26d1-e6bb-4f8e-8b33-3c9277139663","Type":"ContainerStarted","Data":"b412372105a42b0dd1ef0d75a4d382afd3e5b88cb2ad7b82f3062f469ee55811"} Nov 02 13:59:59.827650 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:59.827210 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-687865856c-95sjw" event={"ID":"605e26d1-e6bb-4f8e-8b33-3c9277139663","Type":"ContainerStarted","Data":"6027abd04663af8f4e6d9b613f4971d71dc60fe1b848d2710d534551e1228cfc"} Nov 02 13:59:59.875209 ip-10-0-134-9 kubenswrapper[4431]: I1102 13:59:59.875163 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-687865856c-95sjw" podStartSLOduration=1.875150007 podStartE2EDuration="1.875150007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:59.874571279 +0000 UTC m=+164.911676226" watchObservedRunningTime="2026-11-02 13:59:59.875150007 +0000 UTC m=+164.912255026" Nov 02 14:00:08.841860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:08.841829 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-687865856c-95sjw" Nov 02 14:00:08.842202 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:08.841883 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-687865856c-95sjw" Nov 02 14:00:08.846685 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:08.846665 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-687865856c-95sjw" Nov 02 14:00:08.864519 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:08.864500 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-687865856c-95sjw" Nov 02 14:00:08.979844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:08.979816 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74468c9945-7qj7c"] Nov 02 14:00:21.281353 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:00:21.281315 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:00:34.003542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.003505 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-74468c9945-7qj7c" podUID="4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" containerName="console" containerID="cri-o://5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971" gracePeriod=15 Nov 02 14:00:34.247108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.247082 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74468c9945-7qj7c" Nov 02 14:00:34.353326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353266 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-serving-cert\") pod \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " Nov 02 14:00:34.353326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353297 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-trusted-ca-bundle\") pod \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " Nov 02 14:00:34.353484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353336 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-config\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-config\") pod \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " Nov 02 14:00:34.353484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353377 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-oauth-serving-cert\") pod \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " Nov 02 14:00:34.353484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353407 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-tmp\") pod \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " Nov 02 14:00:34.353484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353444 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-oauth-config\") pod \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " Nov 02 14:00:34.353671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353491 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-service-ca\") pod \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " Nov 02 14:00:34.353671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353520 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-kube-api-access-qdwmt\" (UniqueName: \"kubernetes.io/projected/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-kube-api-access-qdwmt\") pod \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\" (UID: \"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f\") " Nov 02 14:00:34.353798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353719 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-tmp" pod "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" (UID: "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:00:34.353798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353747 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-trusted-ca-bundle" pod "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" (UID: "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:00:34.353925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353813 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-config" pod "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" (UID: "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:00:34.353925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.353831 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-oauth-serving-cert" pod "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" (UID: "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:00:34.354249 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.354222 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-service-ca" pod "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" (UID: "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:00:34.355428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.355408 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-serving-cert" pod "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" (UID: "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:00:34.355521 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.355502 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-oauth-config" pod "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" (UID: "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:00:34.355699 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.355681 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-kube-api-access-qdwmt" pod "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" (UID: "4aa9d0eb-6097-4ddf-968a-2e085fb2a45f"). InnerVolumeSpecName "kube-api-access-qdwmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:00:34.454504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.454482 4431 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-serving-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:00:34.454504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.454501 4431 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-trusted-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:00:34.454645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.454510 4431 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:00:34.454645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.454519 4431 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-oauth-serving-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:00:34.454645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.454527 4431 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-tmp\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:00:34.454645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.454535 4431 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-console-oauth-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:00:34.454645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.454543 4431 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:00:34.454645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.454551 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdwmt\" (UniqueName: \"kubernetes.io/projected/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f-kube-api-access-qdwmt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:00:34.959765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.959735 4431 generic.go:356] "Generic (PLEG): container finished" podID="4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" containerID="5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971" exitCode=0 Nov 02 14:00:34.959967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.959809 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74468c9945-7qj7c" Nov 02 14:00:34.959967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.959828 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74468c9945-7qj7c" event={"ID":"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f","Type":"ContainerDied","Data":"5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971"} Nov 02 14:00:34.959967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.959912 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74468c9945-7qj7c" event={"ID":"4aa9d0eb-6097-4ddf-968a-2e085fb2a45f","Type":"ContainerDied","Data":"f5dd1d6a3fe3ff9e1d5905fa88994697cc535bab5d88c032cc53e243c8958076"} Nov 02 14:00:34.959967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.959938 4431 scope.go:122] "RemoveContainer" containerID="5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971" Nov 02 14:00:34.968660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.968646 4431 scope.go:122] "RemoveContainer" containerID="5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971" Nov 02 14:00:34.968951 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:00:34.968915 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971\": container with ID starting with 5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971 not found: ID does not exist" containerID="5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971" Nov 02 14:00:34.968999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:34.968959 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971"} err="failed to get container status \"5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971\": rpc error: code = NotFound desc = could not find container \"5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971\": container with ID starting with 5dcc31bf1eff8f08d205983fa2e5095b136495b2711c66ee002a42ed5c951971 not found: ID does not exist" Nov 02 14:00:35.017209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:35.017187 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74468c9945-7qj7c"] Nov 02 14:00:35.029736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:35.029716 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74468c9945-7qj7c"] Nov 02 14:00:35.073382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:00:35.073361 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" path="/var/lib/kubelet/pods/4aa9d0eb-6097-4ddf-968a-2e085fb2a45f/volumes" Nov 02 14:00:51.283254 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:00:51.283210 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:01:21.282203 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:01:21.282159 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:01:51.280512 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:01:51.280468 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:02:21.280529 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:02:21.280494 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:02:51.280609 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:02:51.280574 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:03:16.226527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.226492 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp"] Nov 02 14:03:16.226972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.226824 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" containerName="console" Nov 02 14:03:16.226972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.226837 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" containerName="console" Nov 02 14:03:16.226972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.226896 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa9d0eb-6097-4ddf-968a-2e085fb2a45f" containerName="console" Nov 02 14:03:16.232138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.232111 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.269880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.269854 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp"] Nov 02 14:03:16.302480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.302452 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.302601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.302488 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.302601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.302508 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.302697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.302627 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.302697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.302649 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.302697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.302672 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9shn\" (UniqueName: \"kubernetes.io/projected/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-kube-api-access-f9shn\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.327911 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.327888 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 14:03:16.328001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.327942 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 14:03:16.328001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.327975 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 14:03:16.328104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.328078 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 14:03:16.328188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.328170 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sd8v9\"" Nov 02 14:03:16.328230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.328194 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 14:03:16.403526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.403504 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.403614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.403535 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.403614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.403558 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.403717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.403612 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.403717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.403641 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.403862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.403794 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9shn\" (UniqueName: \"kubernetes.io/projected/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-kube-api-access-f9shn\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.403994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.403973 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-manifests\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.406125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.406105 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-webhook-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.406220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.406186 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.406334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.406313 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-tmp\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.406428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.406413 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-metrics-cert\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.486137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.486082 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9shn\" (UniqueName: \"kubernetes.io/projected/cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290-kube-api-access-f9shn\") pod \"opendatahub-operator-controller-manager-78cdb466dc-d84mp\" (UID: \"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.547544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.547525 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:16.905907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.905868 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp"] Nov 02 14:03:16.909275 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:03:16.909250 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1ff8c9_abc2_4f9c_8f4f_d8f60177e290.slice/crio-b6da16231bb3c068ec4a56d6f6994c8bba077d13a91bae69be08ab2cbf91fa20 WatchSource:0}: Error finding container b6da16231bb3c068ec4a56d6f6994c8bba077d13a91bae69be08ab2cbf91fa20: Status 404 returned error can't find the container with id b6da16231bb3c068ec4a56d6f6994c8bba077d13a91bae69be08ab2cbf91fa20 Nov 02 14:03:16.911140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:16.911122 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:03:17.570066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:17.570033 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" event={"ID":"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290","Type":"ContainerStarted","Data":"b6da16231bb3c068ec4a56d6f6994c8bba077d13a91bae69be08ab2cbf91fa20"} Nov 02 14:03:20.584144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:20.584079 4431 generic.go:356] "Generic (PLEG): container finished" podID="cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290" containerID="404cf65d152401c6e6f4c0cb8cec85958adc461532ece24880625c43e938d989" exitCode=0 Nov 02 14:03:20.584144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:20.584129 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" event={"ID":"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290","Type":"ContainerDied","Data":"404cf65d152401c6e6f4c0cb8cec85958adc461532ece24880625c43e938d989"} Nov 02 14:03:21.282727 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:03:21.282649 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:03:21.589850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:21.589741 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" event={"ID":"cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290","Type":"ContainerStarted","Data":"91264382a2555ae1aade78685058c34ea25075edabf9f5685669ac15cc39544e"} Nov 02 14:03:21.589850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:21.589830 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:21.697528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:21.697477 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" podStartSLOduration=2.080711848 podStartE2EDuration="5.69746281s" totalImagesPullingTime="3.131521603s" totalInitContainerRuntime="485.229359ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:21.683434941 +0000 UTC m=+366.720539873" watchObservedRunningTime="2026-11-02 14:03:21.69746281 +0000 UTC m=+366.734567720" Nov 02 14:03:28.698588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.698547 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp"] Nov 02 14:03:28.701825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.701809 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:28.731367 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.731345 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 14:03:28.731367 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.731363 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 14:03:28.733435 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.733414 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp"] Nov 02 14:03:28.737071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.737055 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tvbb2\"" Nov 02 14:03:28.799572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.799546 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5w64\" (UniqueName: \"kubernetes.io/projected/0d666d72-faf5-4172-bcb5-04db28685143-kube-api-access-x5w64\") pod \"cert-manager-webhook-5dbbbf75cc-mj5lp\" (UID: \"0d666d72-faf5-4172-bcb5-04db28685143\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:28.799669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.799595 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d666d72-faf5-4172-bcb5-04db28685143-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mj5lp\" (UID: \"0d666d72-faf5-4172-bcb5-04db28685143\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:28.900443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.900421 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5w64\" (UniqueName: \"kubernetes.io/projected/0d666d72-faf5-4172-bcb5-04db28685143-kube-api-access-x5w64\") pod \"cert-manager-webhook-5dbbbf75cc-mj5lp\" (UID: \"0d666d72-faf5-4172-bcb5-04db28685143\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:28.900539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.900465 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d666d72-faf5-4172-bcb5-04db28685143-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mj5lp\" (UID: \"0d666d72-faf5-4172-bcb5-04db28685143\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:28.924854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.924827 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5w64\" (UniqueName: \"kubernetes.io/projected/0d666d72-faf5-4172-bcb5-04db28685143-kube-api-access-x5w64\") pod \"cert-manager-webhook-5dbbbf75cc-mj5lp\" (UID: \"0d666d72-faf5-4172-bcb5-04db28685143\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:28.925539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:28.925517 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d666d72-faf5-4172-bcb5-04db28685143-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-mj5lp\" (UID: \"0d666d72-faf5-4172-bcb5-04db28685143\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:29.027935 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:29.027883 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:29.326158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:29.326136 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp"] Nov 02 14:03:29.328363 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:03:29.328335 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d666d72_faf5_4172_bcb5_04db28685143.slice/crio-bff9fbe0501b257b7cb54a23b24118474693b62532784f277abca9c283e490f3 WatchSource:0}: Error finding container bff9fbe0501b257b7cb54a23b24118474693b62532784f277abca9c283e490f3: Status 404 returned error can't find the container with id bff9fbe0501b257b7cb54a23b24118474693b62532784f277abca9c283e490f3 Nov 02 14:03:29.622923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:29.622851 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" event={"ID":"0d666d72-faf5-4172-bcb5-04db28685143","Type":"ContainerStarted","Data":"bff9fbe0501b257b7cb54a23b24118474693b62532784f277abca9c283e490f3"} Nov 02 14:03:32.596605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:32.596537 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78cdb466dc-d84mp" Nov 02 14:03:32.638508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:32.638475 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" event={"ID":"0d666d72-faf5-4172-bcb5-04db28685143","Type":"ContainerStarted","Data":"533b978ad82c083a4251da2903369954c1447457d8d9868db58bbf8e0144cf0c"} Nov 02 14:03:32.638671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:32.638526 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:32.767155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:32.767098 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" podStartSLOduration=1.823901184 podStartE2EDuration="4.767082861s" totalImagesPullingTime="2.943181677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:32.766149465 +0000 UTC m=+377.803254396" watchObservedRunningTime="2026-11-02 14:03:32.767082861 +0000 UTC m=+377.804187792" Nov 02 14:03:36.914046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:36.914010 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g"] Nov 02 14:03:36.919408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:36.919392 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" Nov 02 14:03:36.939587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:36.939565 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g"] Nov 02 14:03:36.951600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:36.951576 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:03:36.951682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:36.951601 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 14:03:36.951682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:36.951640 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-47dv7\"" Nov 02 14:03:37.066866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.066847 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhgr\" (UniqueName: \"kubernetes.io/projected/e2532e4b-b7e8-4a9c-b300-7f008758df4a-kube-api-access-jzhgr\") pod \"jobset-operator-674cd59dc5-fqd5g\" (UID: \"e2532e4b-b7e8-4a9c-b300-7f008758df4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" Nov 02 14:03:37.066955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.066873 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2532e4b-b7e8-4a9c-b300-7f008758df4a-tmp\") pod \"jobset-operator-674cd59dc5-fqd5g\" (UID: \"e2532e4b-b7e8-4a9c-b300-7f008758df4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" Nov 02 14:03:37.167504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.167435 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhgr\" (UniqueName: \"kubernetes.io/projected/e2532e4b-b7e8-4a9c-b300-7f008758df4a-kube-api-access-jzhgr\") pod \"jobset-operator-674cd59dc5-fqd5g\" (UID: \"e2532e4b-b7e8-4a9c-b300-7f008758df4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" Nov 02 14:03:37.167504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.167474 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2532e4b-b7e8-4a9c-b300-7f008758df4a-tmp\") pod \"jobset-operator-674cd59dc5-fqd5g\" (UID: \"e2532e4b-b7e8-4a9c-b300-7f008758df4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" Nov 02 14:03:37.167773 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.167757 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2532e4b-b7e8-4a9c-b300-7f008758df4a-tmp\") pod \"jobset-operator-674cd59dc5-fqd5g\" (UID: \"e2532e4b-b7e8-4a9c-b300-7f008758df4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" Nov 02 14:03:37.195861 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.195833 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhgr\" (UniqueName: \"kubernetes.io/projected/e2532e4b-b7e8-4a9c-b300-7f008758df4a-kube-api-access-jzhgr\") pod \"jobset-operator-674cd59dc5-fqd5g\" (UID: \"e2532e4b-b7e8-4a9c-b300-7f008758df4a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" Nov 02 14:03:37.230081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.230061 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" Nov 02 14:03:37.541140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.541063 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g"] Nov 02 14:03:37.544491 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:03:37.544461 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2532e4b_b7e8_4a9c_b300_7f008758df4a.slice/crio-9f4329898585c2b2a0cb123e0f2b1f42c226be795abbab98184ad05d886330e8 WatchSource:0}: Error finding container 9f4329898585c2b2a0cb123e0f2b1f42c226be795abbab98184ad05d886330e8: Status 404 returned error can't find the container with id 9f4329898585c2b2a0cb123e0f2b1f42c226be795abbab98184ad05d886330e8 Nov 02 14:03:37.659474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:37.659441 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" event={"ID":"e2532e4b-b7e8-4a9c-b300-7f008758df4a","Type":"ContainerStarted","Data":"9f4329898585c2b2a0cb123e0f2b1f42c226be795abbab98184ad05d886330e8"} Nov 02 14:03:38.645293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.645260 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-mj5lp" Nov 02 14:03:38.653462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.653421 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj"] Nov 02 14:03:38.656087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.656062 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.677429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.677408 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 14:03:38.677429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.677423 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 14:03:38.677824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.677803 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cph66\"" Nov 02 14:03:38.677916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.677846 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 14:03:38.679692 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.679668 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj"] Nov 02 14:03:38.774724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.774696 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 14:03:38.781108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.781079 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwfcv\" (UniqueName: \"kubernetes.io/projected/1b17862c-afb6-4a79-a21f-54b8e5ea3063-kube-api-access-vwfcv\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.781242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.781133 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b17862c-afb6-4a79-a21f-54b8e5ea3063-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.781242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.781175 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b17862c-afb6-4a79-a21f-54b8e5ea3063-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.881769 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.881743 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwfcv\" (UniqueName: \"kubernetes.io/projected/1b17862c-afb6-4a79-a21f-54b8e5ea3063-kube-api-access-vwfcv\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.881936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.881795 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b17862c-afb6-4a79-a21f-54b8e5ea3063-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.881936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.881831 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b17862c-afb6-4a79-a21f-54b8e5ea3063-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.882628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.882559 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1b17862c-afb6-4a79-a21f-54b8e5ea3063-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.884401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.884381 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b17862c-afb6-4a79-a21f-54b8e5ea3063-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.910017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.909952 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwfcv\" (UniqueName: \"kubernetes.io/projected/1b17862c-afb6-4a79-a21f-54b8e5ea3063-kube-api-access-vwfcv\") pod \"kuadrant-console-plugin-66c4fb575d-mgjcj\" (UID: \"1b17862c-afb6-4a79-a21f-54b8e5ea3063\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:38.971333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:38.971307 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" Nov 02 14:03:39.289166 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:39.289091 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj"] Nov 02 14:03:39.292773 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:03:39.292738 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b17862c_afb6_4a79_a21f_54b8e5ea3063.slice/crio-88b0227517cb1534dd6deca5d584cfa7905088ddbf119b23df2465bdc84390cd WatchSource:0}: Error finding container 88b0227517cb1534dd6deca5d584cfa7905088ddbf119b23df2465bdc84390cd: Status 404 returned error can't find the container with id 88b0227517cb1534dd6deca5d584cfa7905088ddbf119b23df2465bdc84390cd Nov 02 14:03:39.671295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:39.671262 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" event={"ID":"1b17862c-afb6-4a79-a21f-54b8e5ea3063","Type":"ContainerStarted","Data":"88b0227517cb1534dd6deca5d584cfa7905088ddbf119b23df2465bdc84390cd"} Nov 02 14:03:40.360726 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.360694 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-74djh"] Nov 02 14:03:40.363599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.363581 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-74djh" Nov 02 14:03:40.381749 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.381719 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-v66h7\"" Nov 02 14:03:40.395133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.395107 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-74djh"] Nov 02 14:03:40.497260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.497222 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d57f\" (UniqueName: \"kubernetes.io/projected/3f153a84-0da6-47a5-aaea-26fc51b053ba-kube-api-access-5d57f\") pod \"cert-manager-597b95cb84-74djh\" (UID: \"3f153a84-0da6-47a5-aaea-26fc51b053ba\") " pod="cert-manager/cert-manager-597b95cb84-74djh" Nov 02 14:03:40.497418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.497296 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f153a84-0da6-47a5-aaea-26fc51b053ba-bound-sa-token\") pod \"cert-manager-597b95cb84-74djh\" (UID: \"3f153a84-0da6-47a5-aaea-26fc51b053ba\") " pod="cert-manager/cert-manager-597b95cb84-74djh" Nov 02 14:03:40.598154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.598078 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f153a84-0da6-47a5-aaea-26fc51b053ba-bound-sa-token\") pod \"cert-manager-597b95cb84-74djh\" (UID: \"3f153a84-0da6-47a5-aaea-26fc51b053ba\") " pod="cert-manager/cert-manager-597b95cb84-74djh" Nov 02 14:03:40.598296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.598163 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d57f\" (UniqueName: \"kubernetes.io/projected/3f153a84-0da6-47a5-aaea-26fc51b053ba-kube-api-access-5d57f\") pod \"cert-manager-597b95cb84-74djh\" (UID: \"3f153a84-0da6-47a5-aaea-26fc51b053ba\") " pod="cert-manager/cert-manager-597b95cb84-74djh" Nov 02 14:03:40.626739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.626711 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f153a84-0da6-47a5-aaea-26fc51b053ba-bound-sa-token\") pod \"cert-manager-597b95cb84-74djh\" (UID: \"3f153a84-0da6-47a5-aaea-26fc51b053ba\") " pod="cert-manager/cert-manager-597b95cb84-74djh" Nov 02 14:03:40.626890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.626759 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d57f\" (UniqueName: \"kubernetes.io/projected/3f153a84-0da6-47a5-aaea-26fc51b053ba-kube-api-access-5d57f\") pod \"cert-manager-597b95cb84-74djh\" (UID: \"3f153a84-0da6-47a5-aaea-26fc51b053ba\") " pod="cert-manager/cert-manager-597b95cb84-74djh" Nov 02 14:03:40.679462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.679428 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" event={"ID":"e2532e4b-b7e8-4a9c-b300-7f008758df4a","Type":"ContainerStarted","Data":"4e820d2f3552e82d3cc289a32ebed7e9853de7a56e2396cf2bbf831932fcfa8c"} Nov 02 14:03:40.711120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.711092 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-74djh" Nov 02 14:03:40.733029 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:40.732974 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fqd5g" podStartSLOduration=1.9378558030000002 podStartE2EDuration="4.732956369s" totalImagesPullingTime="2.795100566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:40.731583471 +0000 UTC m=+385.768688403" watchObservedRunningTime="2026-11-02 14:03:40.732956369 +0000 UTC m=+385.770061302" Nov 02 14:03:41.046561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:41.046530 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-74djh"] Nov 02 14:03:41.049540 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:03:41.049510 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f153a84_0da6_47a5_aaea_26fc51b053ba.slice/crio-888d74c2680f09908d95952d5cc37b2c3f4df5dabc74182e48e9c08122583d90 WatchSource:0}: Error finding container 888d74c2680f09908d95952d5cc37b2c3f4df5dabc74182e48e9c08122583d90: Status 404 returned error can't find the container with id 888d74c2680f09908d95952d5cc37b2c3f4df5dabc74182e48e9c08122583d90 Nov 02 14:03:41.685020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:41.684985 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-74djh" event={"ID":"3f153a84-0da6-47a5-aaea-26fc51b053ba","Type":"ContainerStarted","Data":"2b01966cc014a0b2e54426a6e4e72704cfa548f91739aac7e3e2d25d072ed591"} Nov 02 14:03:41.685391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:41.685030 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-74djh" event={"ID":"3f153a84-0da6-47a5-aaea-26fc51b053ba","Type":"ContainerStarted","Data":"888d74c2680f09908d95952d5cc37b2c3f4df5dabc74182e48e9c08122583d90"} Nov 02 14:03:41.730694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:41.730644 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-74djh" podStartSLOduration=1.730631787 podStartE2EDuration="1.730631787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:41.729427229 +0000 UTC m=+386.766532161" watchObservedRunningTime="2026-11-02 14:03:41.730631787 +0000 UTC m=+386.767736718" Nov 02 14:03:44.702277 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:44.702250 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" event={"ID":"1b17862c-afb6-4a79-a21f-54b8e5ea3063","Type":"ContainerStarted","Data":"b90464e1608919753146bf5c32412c4872aa103c14ef1d3bba5d543fc60a409c"} Nov 02 14:03:51.281062 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:03:51.281024 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:03:53.271954 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.271897 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mgjcj" podStartSLOduration=9.948099946 podStartE2EDuration="15.271880324s" totalImagesPullingTime="5.323780378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:44.757353608 +0000 UTC m=+389.794458541" watchObservedRunningTime="2026-11-02 14:03:53.271880324 +0000 UTC m=+398.308985257" Nov 02 14:03:53.272570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.272553 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d"] Nov 02 14:03:53.277098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.277080 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.297210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.297189 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 14:03:53.298873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.298841 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d"] Nov 02 14:03:53.306352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.306332 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 14:03:53.310364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.310343 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j28bz\"" Nov 02 14:03:53.310458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.310427 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 14:03:53.310458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.310436 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 14:03:53.311686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.311667 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxtk\" (UniqueName: \"kubernetes.io/projected/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-kube-api-access-8fxtk\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.311740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.311725 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-tls-certs\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.311855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.311837 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-tmp\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.412649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.412623 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-tls-certs\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.412649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.412650 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-tmp\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.412814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.412684 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxtk\" (UniqueName: \"kubernetes.io/projected/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-kube-api-access-8fxtk\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.414902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.414881 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-tmp\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.415054 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.415037 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-tls-certs\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.442164 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.442138 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxtk\" (UniqueName: \"kubernetes.io/projected/33771e58-bf16-4c4d-bb9c-ac49aa5d26df-kube-api-access-8fxtk\") pod \"kube-auth-proxy-8f9c48f5f-prw2d\" (UID: \"33771e58-bf16-4c4d-bb9c-ac49aa5d26df\") " pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.589932 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.589857 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" Nov 02 14:03:53.889643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:53.889614 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d"] Nov 02 14:03:53.892371 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:03:53.892340 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33771e58_bf16_4c4d_bb9c_ac49aa5d26df.slice/crio-70e3ca92a67bf3dd6b467255179c55d40030d485ae2e7edf795194b27ade634e WatchSource:0}: Error finding container 70e3ca92a67bf3dd6b467255179c55d40030d485ae2e7edf795194b27ade634e: Status 404 returned error can't find the container with id 70e3ca92a67bf3dd6b467255179c55d40030d485ae2e7edf795194b27ade634e Nov 02 14:03:54.746604 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:54.746560 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" event={"ID":"33771e58-bf16-4c4d-bb9c-ac49aa5d26df","Type":"ContainerStarted","Data":"70e3ca92a67bf3dd6b467255179c55d40030d485ae2e7edf795194b27ade634e"} Nov 02 14:03:56.757042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:56.757001 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" event={"ID":"33771e58-bf16-4c4d-bb9c-ac49aa5d26df","Type":"ContainerStarted","Data":"df063fd78c6a58b023483e378ffbdb209c1dae5228a255d844cc0cb8862cf262"} Nov 02 14:03:56.804000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:03:56.803948 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8f9c48f5f-prw2d" podStartSLOduration=1.398657346 podStartE2EDuration="3.803934679s" totalImagesPullingTime="2.405277333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:56.802336188 +0000 UTC m=+401.839441121" watchObservedRunningTime="2026-11-02 14:03:56.803934679 +0000 UTC m=+401.841039610" Nov 02 14:04:15.102859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:15.102829 4431 scope.go:122] "RemoveContainer" containerID="3f28c2e7975088c0c85e6e9ab7acb15ff6e11d2e75aaaad961734d362c0c6602" Nov 02 14:04:15.110717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:15.110694 4431 scope.go:122] "RemoveContainer" containerID="f743a84796ba585390512a199a30466a3952b620827f02944580b1ee1c51aa85" Nov 02 14:04:15.117700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:15.117677 4431 scope.go:122] "RemoveContainer" containerID="ebb10bab3010feff012ec3b3c60486371e1ea3ab1fc6bc4a4eb0a63eb681fb81" Nov 02 14:04:15.124205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:15.124189 4431 scope.go:122] "RemoveContainer" containerID="353a7c61c845265ce519869588cc41db67bb81db679ccb6491c8d4b92737141d" Nov 02 14:04:15.131519 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:15.131504 4431 scope.go:122] "RemoveContainer" containerID="b190ab69b25c5abea4b30ac71425ede143738a88dc5495711968e4489e0156ab" Nov 02 14:04:15.138003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:15.137988 4431 scope.go:122] "RemoveContainer" containerID="f4d454c5cd40889173fd7b41d8e0e7f3dba1e87ae568c0d294a7db799414814f" Nov 02 14:04:15.144609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:15.144592 4431 scope.go:122] "RemoveContainer" containerID="1cb32360504b3752f6b285df414c6c58f51c5bae346e2843f8b62851b9a4f9fd" Nov 02 14:04:20.727444 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.727414 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9zhx5"] Nov 02 14:04:20.752926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.752900 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:20.758305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.758275 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9zhx5"] Nov 02 14:04:20.769875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.769839 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 14:04:20.836389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.836356 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b94b6d76-e4d5-4114-bf28-117b97394ec5-config-file\") pod \"limitador-limitador-74dfcfb894-9zhx5\" (UID: \"b94b6d76-e4d5-4114-bf28-117b97394ec5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:20.836561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.836399 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpm99\" (UniqueName: \"kubernetes.io/projected/b94b6d76-e4d5-4114-bf28-117b97394ec5-kube-api-access-tpm99\") pod \"limitador-limitador-74dfcfb894-9zhx5\" (UID: \"b94b6d76-e4d5-4114-bf28-117b97394ec5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:20.850752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.850723 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9zhx5"] Nov 02 14:04:20.937312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.937287 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b94b6d76-e4d5-4114-bf28-117b97394ec5-config-file\") pod \"limitador-limitador-74dfcfb894-9zhx5\" (UID: \"b94b6d76-e4d5-4114-bf28-117b97394ec5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:20.937446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.937335 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpm99\" (UniqueName: \"kubernetes.io/projected/b94b6d76-e4d5-4114-bf28-117b97394ec5-kube-api-access-tpm99\") pod \"limitador-limitador-74dfcfb894-9zhx5\" (UID: \"b94b6d76-e4d5-4114-bf28-117b97394ec5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:20.938090 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.938067 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b94b6d76-e4d5-4114-bf28-117b97394ec5-config-file\") pod \"limitador-limitador-74dfcfb894-9zhx5\" (UID: \"b94b6d76-e4d5-4114-bf28-117b97394ec5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:20.966573 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:20.966549 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpm99\" (UniqueName: \"kubernetes.io/projected/b94b6d76-e4d5-4114-bf28-117b97394ec5-kube-api-access-tpm99\") pod \"limitador-limitador-74dfcfb894-9zhx5\" (UID: \"b94b6d76-e4d5-4114-bf28-117b97394ec5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:21.067826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:21.067744 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:21.280688 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:04:21.280656 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:04:21.380895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:21.380822 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9zhx5"] Nov 02 14:04:21.384052 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:04:21.384023 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94b6d76_e4d5_4114_bf28_117b97394ec5.slice/crio-0833b2af5b525bcb138ff3cbe1bc0591080635e43a0b8a62a5598ca498ebaa92 WatchSource:0}: Error finding container 0833b2af5b525bcb138ff3cbe1bc0591080635e43a0b8a62a5598ca498ebaa92: Status 404 returned error can't find the container with id 0833b2af5b525bcb138ff3cbe1bc0591080635e43a0b8a62a5598ca498ebaa92 Nov 02 14:04:21.872683 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:21.872645 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" event={"ID":"b94b6d76-e4d5-4114-bf28-117b97394ec5","Type":"ContainerStarted","Data":"0833b2af5b525bcb138ff3cbe1bc0591080635e43a0b8a62a5598ca498ebaa92"} Nov 02 14:04:22.579371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.579334 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:04:22.582968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.582938 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:22.604815 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.604788 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:04:22.653604 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.653570 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8lx\" (UniqueName: \"kubernetes.io/projected/4dd71c7a-a298-456d-8d7e-0b1884d987ed-kube-api-access-qw8lx\") pod \"limitador-limitador-7c9b87d574-ghltt\" (UID: \"4dd71c7a-a298-456d-8d7e-0b1884d987ed\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:22.653745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.653626 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4dd71c7a-a298-456d-8d7e-0b1884d987ed-config-file\") pod \"limitador-limitador-7c9b87d574-ghltt\" (UID: \"4dd71c7a-a298-456d-8d7e-0b1884d987ed\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:22.671314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.671251 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:04:22.754314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.754289 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8lx\" (UniqueName: \"kubernetes.io/projected/4dd71c7a-a298-456d-8d7e-0b1884d987ed-kube-api-access-qw8lx\") pod \"limitador-limitador-7c9b87d574-ghltt\" (UID: \"4dd71c7a-a298-456d-8d7e-0b1884d987ed\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:22.754456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.754342 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4dd71c7a-a298-456d-8d7e-0b1884d987ed-config-file\") pod \"limitador-limitador-7c9b87d574-ghltt\" (UID: \"4dd71c7a-a298-456d-8d7e-0b1884d987ed\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:22.755114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.755094 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4dd71c7a-a298-456d-8d7e-0b1884d987ed-config-file\") pod \"limitador-limitador-7c9b87d574-ghltt\" (UID: \"4dd71c7a-a298-456d-8d7e-0b1884d987ed\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:22.780029 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.779999 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8lx\" (UniqueName: \"kubernetes.io/projected/4dd71c7a-a298-456d-8d7e-0b1884d987ed-kube-api-access-qw8lx\") pod \"limitador-limitador-7c9b87d574-ghltt\" (UID: \"4dd71c7a-a298-456d-8d7e-0b1884d987ed\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:22.899556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:22.899532 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:23.206767 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:23.206693 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:04:23.210406 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:04:23.210381 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd71c7a_a298_456d_8d7e_0b1884d987ed.slice/crio-b5c27daa23a7c8586f974663ba115d22f0580c9f622459e40fa40a8436ed5fdf WatchSource:0}: Error finding container b5c27daa23a7c8586f974663ba115d22f0580c9f622459e40fa40a8436ed5fdf: Status 404 returned error can't find the container with id b5c27daa23a7c8586f974663ba115d22f0580c9f622459e40fa40a8436ed5fdf Nov 02 14:04:23.883518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:23.883485 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" event={"ID":"4dd71c7a-a298-456d-8d7e-0b1884d987ed","Type":"ContainerStarted","Data":"b5c27daa23a7c8586f974663ba115d22f0580c9f622459e40fa40a8436ed5fdf"} Nov 02 14:04:24.889684 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:24.889635 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" event={"ID":"b94b6d76-e4d5-4114-bf28-117b97394ec5","Type":"ContainerStarted","Data":"808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc"} Nov 02 14:04:24.890139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:24.889796 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:24.891847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:24.891813 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" event={"ID":"4dd71c7a-a298-456d-8d7e-0b1884d987ed","Type":"ContainerStarted","Data":"b7973a1f710db22f1daccbf1070e851b4e09129b3390f14d7ba8d26354a4a8c1"} Nov 02 14:04:24.891998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:24.891895 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:24.943086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:24.943036 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" podStartSLOduration=2.324473664 podStartE2EDuration="4.943018274s" totalImagesPullingTime="2.61854461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:24.940482147 +0000 UTC m=+429.977587078" watchObservedRunningTime="2026-11-02 14:04:24.943018274 +0000 UTC m=+429.980123207" Nov 02 14:04:24.992576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:24.992530 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" podStartSLOduration=2.082514799 podStartE2EDuration="2.992518988s" totalImagesPullingTime="910.004189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:24.992242627 +0000 UTC m=+430.029347560" watchObservedRunningTime="2026-11-02 14:04:24.992518988 +0000 UTC m=+430.029623919" Nov 02 14:04:35.896725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:35.896694 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-ghltt" Nov 02 14:04:35.897198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:35.896767 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:35.971890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:35.971861 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9zhx5"] Nov 02 14:04:35.972035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:35.972016 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" podUID="b94b6d76-e4d5-4114-bf28-117b97394ec5" containerName="limitador" containerID="cri-o://808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc" gracePeriod=30 Nov 02 14:04:36.513333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.513314 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:36.660200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.660174 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94b6d76-e4d5-4114-bf28-117b97394ec5-kube-api-access-tpm99\" (UniqueName: \"kubernetes.io/projected/b94b6d76-e4d5-4114-bf28-117b97394ec5-kube-api-access-tpm99\") pod \"b94b6d76-e4d5-4114-bf28-117b97394ec5\" (UID: \"b94b6d76-e4d5-4114-bf28-117b97394ec5\") " Nov 02 14:04:36.660307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.660281 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94b6d76-e4d5-4114-bf28-117b97394ec5-config-file\" (UniqueName: \"kubernetes.io/configmap/b94b6d76-e4d5-4114-bf28-117b97394ec5-config-file\") pod \"b94b6d76-e4d5-4114-bf28-117b97394ec5\" (UID: \"b94b6d76-e4d5-4114-bf28-117b97394ec5\") " Nov 02 14:04:36.660612 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.660585 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94b6d76-e4d5-4114-bf28-117b97394ec5-config-file" pod "b94b6d76-e4d5-4114-bf28-117b97394ec5" (UID: "b94b6d76-e4d5-4114-bf28-117b97394ec5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:04:36.662197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.662173 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94b6d76-e4d5-4114-bf28-117b97394ec5-kube-api-access-tpm99" pod "b94b6d76-e4d5-4114-bf28-117b97394ec5" (UID: "b94b6d76-e4d5-4114-bf28-117b97394ec5"). InnerVolumeSpecName "kube-api-access-tpm99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:04:36.761679 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.761652 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpm99\" (UniqueName: \"kubernetes.io/projected/b94b6d76-e4d5-4114-bf28-117b97394ec5-kube-api-access-tpm99\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:04:36.761679 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.761676 4431 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b94b6d76-e4d5-4114-bf28-117b97394ec5-config-file\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:04:36.942376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.942308 4431 generic.go:356] "Generic (PLEG): container finished" podID="b94b6d76-e4d5-4114-bf28-117b97394ec5" containerID="808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc" exitCode=0 Nov 02 14:04:36.942376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.942351 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" Nov 02 14:04:36.942376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.942352 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" event={"ID":"b94b6d76-e4d5-4114-bf28-117b97394ec5","Type":"ContainerDied","Data":"808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc"} Nov 02 14:04:36.942860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.942406 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9zhx5" event={"ID":"b94b6d76-e4d5-4114-bf28-117b97394ec5","Type":"ContainerDied","Data":"0833b2af5b525bcb138ff3cbe1bc0591080635e43a0b8a62a5598ca498ebaa92"} Nov 02 14:04:36.942860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.942429 4431 scope.go:122] "RemoveContainer" containerID="808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc" Nov 02 14:04:36.950719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.950702 4431 scope.go:122] "RemoveContainer" containerID="808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc" Nov 02 14:04:36.950974 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:04:36.950959 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc\": container with ID starting with 808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc not found: ID does not exist" containerID="808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc" Nov 02 14:04:36.951026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.950982 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc"} err="failed to get container status \"808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc\": rpc error: code = NotFound desc = could not find container \"808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc\": container with ID starting with 808ece7c0b3decc541b5a224a7c584621d1e09067fb238c6a570436faf0841bc not found: ID does not exist" Nov 02 14:04:36.997242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:36.997222 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9zhx5"] Nov 02 14:04:37.010227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:37.010206 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9zhx5"] Nov 02 14:04:37.073545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:37.073510 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94b6d76-e4d5-4114-bf28-117b97394ec5" path="/var/lib/kubelet/pods/b94b6d76-e4d5-4114-bf28-117b97394ec5/volumes" Nov 02 14:04:41.590190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.590156 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk"] Nov 02 14:04:41.590649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.590480 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b94b6d76-e4d5-4114-bf28-117b97394ec5" containerName="limitador" Nov 02 14:04:41.590649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.590491 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94b6d76-e4d5-4114-bf28-117b97394ec5" containerName="limitador" Nov 02 14:04:41.590649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.590569 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94b6d76-e4d5-4114-bf28-117b97394ec5" containerName="limitador" Nov 02 14:04:41.595093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.595072 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:41.612834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.612814 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kt5kt\"" Nov 02 14:04:41.614067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.614043 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk"] Nov 02 14:04:41.625946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.625906 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 14:04:41.626070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.625995 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 14:04:41.626070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.626034 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 14:04:41.701948 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.701927 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5665c19f-2391-4df6-b139-a7021a3285c8-data\") pod \"postgres-7cdbf6dc44-6x7qk\" (UID: \"5665c19f-2391-4df6-b139-a7021a3285c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:41.702050 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.701983 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zlnk\" (UniqueName: \"kubernetes.io/projected/5665c19f-2391-4df6-b139-a7021a3285c8-kube-api-access-9zlnk\") pod \"postgres-7cdbf6dc44-6x7qk\" (UID: \"5665c19f-2391-4df6-b139-a7021a3285c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:41.802477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.802449 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zlnk\" (UniqueName: \"kubernetes.io/projected/5665c19f-2391-4df6-b139-a7021a3285c8-kube-api-access-9zlnk\") pod \"postgres-7cdbf6dc44-6x7qk\" (UID: \"5665c19f-2391-4df6-b139-a7021a3285c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:41.802586 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.802512 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5665c19f-2391-4df6-b139-a7021a3285c8-data\") pod \"postgres-7cdbf6dc44-6x7qk\" (UID: \"5665c19f-2391-4df6-b139-a7021a3285c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:41.802837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.802819 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5665c19f-2391-4df6-b139-a7021a3285c8-data\") pod \"postgres-7cdbf6dc44-6x7qk\" (UID: \"5665c19f-2391-4df6-b139-a7021a3285c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:41.827526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.827496 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zlnk\" (UniqueName: \"kubernetes.io/projected/5665c19f-2391-4df6-b139-a7021a3285c8-kube-api-access-9zlnk\") pod \"postgres-7cdbf6dc44-6x7qk\" (UID: \"5665c19f-2391-4df6-b139-a7021a3285c8\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:41.907476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:41.907457 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:42.202026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:42.201955 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk"] Nov 02 14:04:42.204832 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:04:42.204807 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5665c19f_2391_4df6_b139_a7021a3285c8.slice/crio-f40c47107c54143f8640d7854b63cbd20b8fc95434ff15f80c05a78eee5ee926 WatchSource:0}: Error finding container f40c47107c54143f8640d7854b63cbd20b8fc95434ff15f80c05a78eee5ee926: Status 404 returned error can't find the container with id f40c47107c54143f8640d7854b63cbd20b8fc95434ff15f80c05a78eee5ee926 Nov 02 14:04:42.972119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:42.972074 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" event={"ID":"5665c19f-2391-4df6-b139-a7021a3285c8","Type":"ContainerStarted","Data":"f40c47107c54143f8640d7854b63cbd20b8fc95434ff15f80c05a78eee5ee926"} Nov 02 14:04:46.993300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:46.993270 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" event={"ID":"5665c19f-2391-4df6-b139-a7021a3285c8","Type":"ContainerStarted","Data":"2df18d22fb0221303c2ea36dcdcba41577cd38c170c10605e051362b9c178349"} Nov 02 14:04:46.993574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:46.993364 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:04:47.037650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:47.037608 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" podStartSLOduration=1.323127003 podStartE2EDuration="6.037596351s" totalImagesPullingTime="4.714469348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:47.037344188 +0000 UTC m=+452.074449121" watchObservedRunningTime="2026-11-02 14:04:47.037596351 +0000 UTC m=+452.074701282" Nov 02 14:04:51.284602 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:04:51.284569 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:04:53.026714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:04:53.026689 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-6x7qk" Nov 02 14:05:00.334167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.334129 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7xw9r"] Nov 02 14:05:00.337424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.337408 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:00.359143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.359121 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7xw9r"] Nov 02 14:05:00.367267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.367249 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 14:05:00.367359 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.367312 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 14:05:00.401512 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.401490 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv"] Nov 02 14:05:00.404619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.404603 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:00.427293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.427275 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 14:05:00.427430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.427417 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jj74l\"" Nov 02 14:05:00.427528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.427515 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-l9vhp\"" Nov 02 14:05:00.440153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.440130 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv"] Nov 02 14:05:00.449060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.449041 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4kw4d"] Nov 02 14:05:00.449835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.449814 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpk4\" (UniqueName: \"kubernetes.io/projected/49b694a0-4451-451b-b77b-f2c1774f478f-kube-api-access-nvpk4\") pod \"kserve-controller-manager-776cc66d47-7xw9r\" (UID: \"49b694a0-4451-451b-b77b-f2c1774f478f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:00.449926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.449893 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49b694a0-4451-451b-b77b-f2c1774f478f-cert\") pod \"kserve-controller-manager-776cc66d47-7xw9r\" (UID: \"49b694a0-4451-451b-b77b-f2c1774f478f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:00.452646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.452632 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:00.459566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.459547 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 14:05:00.475159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.475145 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 14:05:00.477340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.477320 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4kw4d"] Nov 02 14:05:00.493469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.493451 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-z2nb2"] Nov 02 14:05:00.496560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.496544 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.515902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.515879 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pbj2g\"" Nov 02 14:05:00.530116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.530091 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-z2nb2"] Nov 02 14:05:00.532856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.532839 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 14:05:00.533157 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.533141 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vbkvq\"" Nov 02 14:05:00.533224 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.533167 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 14:05:00.551288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.551269 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49b694a0-4451-451b-b77b-f2c1774f478f-cert\") pod \"kserve-controller-manager-776cc66d47-7xw9r\" (UID: \"49b694a0-4451-451b-b77b-f2c1774f478f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:00.551373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.551297 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23fc9928-3865-499b-99f0-4e5ec1811f63-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vqsrv\" (UID: \"23fc9928-3865-499b-99f0-4e5ec1811f63\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:00.551373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.551329 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8w28\" (UniqueName: \"kubernetes.io/projected/e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550-kube-api-access-r8w28\") pod \"model-serving-api-75bd89fdfc-4kw4d\" (UID: \"e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550\") " pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:00.551373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.551351 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpk4\" (UniqueName: \"kubernetes.io/projected/49b694a0-4451-451b-b77b-f2c1774f478f-kube-api-access-nvpk4\") pod \"kserve-controller-manager-776cc66d47-7xw9r\" (UID: \"49b694a0-4451-451b-b77b-f2c1774f478f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:00.551555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.551407 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8zl\" (UniqueName: \"kubernetes.io/projected/23fc9928-3865-499b-99f0-4e5ec1811f63-kube-api-access-lw8zl\") pod \"llmisvc-controller-manager-869d5dcc5d-vqsrv\" (UID: \"23fc9928-3865-499b-99f0-4e5ec1811f63\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:00.551555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.551432 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550-tls-certs\") pod \"model-serving-api-75bd89fdfc-4kw4d\" (UID: \"e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550\") " pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:00.553698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.553680 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49b694a0-4451-451b-b77b-f2c1774f478f-cert\") pod \"kserve-controller-manager-776cc66d47-7xw9r\" (UID: \"49b694a0-4451-451b-b77b-f2c1774f478f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:00.579479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.579453 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpk4\" (UniqueName: \"kubernetes.io/projected/49b694a0-4451-451b-b77b-f2c1774f478f-kube-api-access-nvpk4\") pod \"kserve-controller-manager-776cc66d47-7xw9r\" (UID: \"49b694a0-4451-451b-b77b-f2c1774f478f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:00.649894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.649876 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:00.652190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.652172 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23fc9928-3865-499b-99f0-4e5ec1811f63-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vqsrv\" (UID: \"23fc9928-3865-499b-99f0-4e5ec1811f63\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:00.652241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.652201 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb8c30b8-0631-48c2-8e5e-f7feae79214c-cert\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.652275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.652228 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmjp\" (UniqueName: \"kubernetes.io/projected/bb8c30b8-0631-48c2-8e5e-f7feae79214c-kube-api-access-hvmjp\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.652315 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.652270 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8w28\" (UniqueName: \"kubernetes.io/projected/e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550-kube-api-access-r8w28\") pod \"model-serving-api-75bd89fdfc-4kw4d\" (UID: \"e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550\") " pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:00.652361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.652319 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c30b8-0631-48c2-8e5e-f7feae79214c-metrics-certs\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.652437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.652418 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8zl\" (UniqueName: \"kubernetes.io/projected/23fc9928-3865-499b-99f0-4e5ec1811f63-kube-api-access-lw8zl\") pod \"llmisvc-controller-manager-869d5dcc5d-vqsrv\" (UID: \"23fc9928-3865-499b-99f0-4e5ec1811f63\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:00.652487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.652459 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550-tls-certs\") pod \"model-serving-api-75bd89fdfc-4kw4d\" (UID: \"e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550\") " pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:00.654608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.654584 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23fc9928-3865-499b-99f0-4e5ec1811f63-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vqsrv\" (UID: \"23fc9928-3865-499b-99f0-4e5ec1811f63\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:00.654843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.654822 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550-tls-certs\") pod \"model-serving-api-75bd89fdfc-4kw4d\" (UID: \"e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550\") " pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:00.677366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.677344 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8zl\" (UniqueName: \"kubernetes.io/projected/23fc9928-3865-499b-99f0-4e5ec1811f63-kube-api-access-lw8zl\") pod \"llmisvc-controller-manager-869d5dcc5d-vqsrv\" (UID: \"23fc9928-3865-499b-99f0-4e5ec1811f63\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:00.677845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.677824 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8w28\" (UniqueName: \"kubernetes.io/projected/e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550-kube-api-access-r8w28\") pod \"model-serving-api-75bd89fdfc-4kw4d\" (UID: \"e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550\") " pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:00.723071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.717477 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:00.754446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.753420 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmjp\" (UniqueName: \"kubernetes.io/projected/bb8c30b8-0631-48c2-8e5e-f7feae79214c-kube-api-access-hvmjp\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.754446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.753474 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c30b8-0631-48c2-8e5e-f7feae79214c-metrics-certs\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.754446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.753573 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb8c30b8-0631-48c2-8e5e-f7feae79214c-cert\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.762571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.762530 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb8c30b8-0631-48c2-8e5e-f7feae79214c-cert\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.762571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.762530 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb8c30b8-0631-48c2-8e5e-f7feae79214c-metrics-certs\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.765561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.765376 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:00.780254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.780199 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmjp\" (UniqueName: \"kubernetes.io/projected/bb8c30b8-0631-48c2-8e5e-f7feae79214c-kube-api-access-hvmjp\") pod \"odh-model-controller-769875db56-z2nb2\" (UID: \"bb8c30b8-0631-48c2-8e5e-f7feae79214c\") " pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.808178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.807844 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:00.874151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.874124 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64d68d6d56-76wsv"] Nov 02 14:05:00.878017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.878000 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:00.897121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.897094 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 14:05:00.905136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.898137 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ftwbw\"" Nov 02 14:05:00.905136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.898996 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 14:05:00.905136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.900862 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 14:05:00.905136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.903878 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d68d6d56-76wsv"] Nov 02 14:05:00.952879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:00.952855 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7xw9r"] Nov 02 14:05:00.954749 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:05:00.954725 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b694a0_4451_451b_b77b_f2c1774f478f.slice/crio-de106dc9045459a1d5a502bc85cceddb7d183881ead4ea0498b2668139df919d WatchSource:0}: Error finding container de106dc9045459a1d5a502bc85cceddb7d183881ead4ea0498b2668139df919d: Status 404 returned error can't find the container with id de106dc9045459a1d5a502bc85cceddb7d183881ead4ea0498b2668139df919d Nov 02 14:05:01.055062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.055037 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-metrics-certs\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.055193 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.055162 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-webhook-cert\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.055246 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.055201 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhf6\" (UniqueName: \"kubernetes.io/projected/30bc48be-afdc-4155-abd0-d81f94772f74-kube-api-access-tqhf6\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.055607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.055586 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" event={"ID":"49b694a0-4451-451b-b77b-f2c1774f478f","Type":"ContainerStarted","Data":"de106dc9045459a1d5a502bc85cceddb7d183881ead4ea0498b2668139df919d"} Nov 02 14:05:01.074444 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.074416 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv"] Nov 02 14:05:01.076767 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:05:01.076747 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23fc9928_3865_499b_99f0_4e5ec1811f63.slice/crio-b4e09547fafccbc0f5b9b0c4a3fb6e83e07f57cf7f58df8cae5aa78f98c8263e WatchSource:0}: Error finding container b4e09547fafccbc0f5b9b0c4a3fb6e83e07f57cf7f58df8cae5aa78f98c8263e: Status 404 returned error can't find the container with id b4e09547fafccbc0f5b9b0c4a3fb6e83e07f57cf7f58df8cae5aa78f98c8263e Nov 02 14:05:01.089151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.089130 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4kw4d"] Nov 02 14:05:01.091550 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:05:01.091529 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d7afae_cb3c_4ae4_b4ae_f6f725ecd550.slice/crio-44ef4f9b00cd0e61f71fe0134b17d6fcb971693aad5c2624e11b7f5547c9df55 WatchSource:0}: Error finding container 44ef4f9b00cd0e61f71fe0134b17d6fcb971693aad5c2624e11b7f5547c9df55: Status 404 returned error can't find the container with id 44ef4f9b00cd0e61f71fe0134b17d6fcb971693aad5c2624e11b7f5547c9df55 Nov 02 14:05:01.141808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.141772 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-z2nb2"] Nov 02 14:05:01.144490 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:05:01.144468 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8c30b8_0631_48c2_8e5e_f7feae79214c.slice/crio-9b61093198651664cafd4115d658e0e11ea774d02a85a3e879c009bc38472273 WatchSource:0}: Error finding container 9b61093198651664cafd4115d658e0e11ea774d02a85a3e879c009bc38472273: Status 404 returned error can't find the container with id 9b61093198651664cafd4115d658e0e11ea774d02a85a3e879c009bc38472273 Nov 02 14:05:01.155572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.155527 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhf6\" (UniqueName: \"kubernetes.io/projected/30bc48be-afdc-4155-abd0-d81f94772f74-kube-api-access-tqhf6\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.155572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.155567 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-metrics-certs\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.155667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.155626 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-webhook-cert\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.158008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.157986 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-metrics-certs\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.158086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.157987 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-webhook-cert\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.180371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.180351 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhf6\" (UniqueName: \"kubernetes.io/projected/30bc48be-afdc-4155-abd0-d81f94772f74-kube-api-access-tqhf6\") pod \"maas-controller-64d68d6d56-76wsv\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.194222 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.194204 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:01.425857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.425767 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-76ms5"] Nov 02 14:05:01.429557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.429536 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.452220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.452195 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-76ms5"] Nov 02 14:05:01.490072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.490046 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64d68d6d56-76wsv"] Nov 02 14:05:01.493326 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:05:01.493298 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bc48be_afdc_4155_abd0_d81f94772f74.slice/crio-f8a56ede4bb10a3f34703ef7c150b59ef83d496cf207192c0b309b2d29fb9e28 WatchSource:0}: Error finding container f8a56ede4bb10a3f34703ef7c150b59ef83d496cf207192c0b309b2d29fb9e28: Status 404 returned error can't find the container with id f8a56ede4bb10a3f34703ef7c150b59ef83d496cf207192c0b309b2d29fb9e28 Nov 02 14:05:01.559453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.559427 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-webhook-cert\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.559564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.559472 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2cdt\" (UniqueName: \"kubernetes.io/projected/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-kube-api-access-k2cdt\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.559653 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.559631 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-metrics-certs\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.660514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.660462 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-metrics-certs\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.660670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.660601 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-webhook-cert\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.660670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.660634 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2cdt\" (UniqueName: \"kubernetes.io/projected/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-kube-api-access-k2cdt\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.663985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.663928 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-metrics-certs\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.665037 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.665013 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-webhook-cert\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.686687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.686629 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2cdt\" (UniqueName: \"kubernetes.io/projected/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-kube-api-access-k2cdt\") pod \"maas-controller-5db8dc9fd8-76ms5\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:01.743049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:01.743014 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:02.069190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:02.069108 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-z2nb2" event={"ID":"bb8c30b8-0631-48c2-8e5e-f7feae79214c","Type":"ContainerStarted","Data":"9b61093198651664cafd4115d658e0e11ea774d02a85a3e879c009bc38472273"} Nov 02 14:05:02.073835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:02.073792 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" event={"ID":"e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550","Type":"ContainerStarted","Data":"44ef4f9b00cd0e61f71fe0134b17d6fcb971693aad5c2624e11b7f5547c9df55"} Nov 02 14:05:02.075552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:02.075527 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d68d6d56-76wsv" event={"ID":"30bc48be-afdc-4155-abd0-d81f94772f74","Type":"ContainerStarted","Data":"f8a56ede4bb10a3f34703ef7c150b59ef83d496cf207192c0b309b2d29fb9e28"} Nov 02 14:05:02.078081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:02.078053 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" event={"ID":"23fc9928-3865-499b-99f0-4e5ec1811f63","Type":"ContainerStarted","Data":"b4e09547fafccbc0f5b9b0c4a3fb6e83e07f57cf7f58df8cae5aa78f98c8263e"} Nov 02 14:05:02.087123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:02.087099 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-76ms5"] Nov 02 14:05:02.090807 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:05:02.090762 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426ccdb4_cca3_4641_8cc1_af9fffbea7f3.slice/crio-477655cddfed35295d0cdbf7020f0df68125fd950ffe495e2c89ddaab09f96ba WatchSource:0}: Error finding container 477655cddfed35295d0cdbf7020f0df68125fd950ffe495e2c89ddaab09f96ba: Status 404 returned error can't find the container with id 477655cddfed35295d0cdbf7020f0df68125fd950ffe495e2c89ddaab09f96ba Nov 02 14:05:03.087320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:03.087287 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" event={"ID":"426ccdb4-cca3-4641-8cc1-af9fffbea7f3","Type":"ContainerStarted","Data":"477655cddfed35295d0cdbf7020f0df68125fd950ffe495e2c89ddaab09f96ba"} Nov 02 14:05:05.104320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:05.104281 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" event={"ID":"49b694a0-4451-451b-b77b-f2c1774f478f","Type":"ContainerStarted","Data":"284de4a459bd8ff5fad6ed2a7ce4e38938b37f5992a76536c02334ab3a0c0d67"} Nov 02 14:05:05.105019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:05.104993 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:05.106865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:05.106745 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" event={"ID":"e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550","Type":"ContainerStarted","Data":"ba4e4ee77b898bc854b7f0d039cff41dbd657632fc3d2cc4b3aaf135d1b43f81"} Nov 02 14:05:05.107002 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:05.106888 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:05.196807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:05.196748 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" podStartSLOduration=1.2542572889999999 podStartE2EDuration="5.196732519s" totalImagesPullingTime="3.94247523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:05.151271779 +0000 UTC m=+470.188376712" watchObservedRunningTime="2026-11-02 14:05:05.196732519 +0000 UTC m=+470.233837451" Nov 02 14:05:06.114469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:06.114148 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d68d6d56-76wsv" event={"ID":"30bc48be-afdc-4155-abd0-d81f94772f74","Type":"ContainerStarted","Data":"0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096"} Nov 02 14:05:06.114469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:06.114218 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:06.117537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:06.117087 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" event={"ID":"23fc9928-3865-499b-99f0-4e5ec1811f63","Type":"ContainerStarted","Data":"93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4"} Nov 02 14:05:06.117537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:06.117343 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:06.186456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:06.185843 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" podStartSLOduration=3.055095279 podStartE2EDuration="6.185823776s" totalImagesPullingTime="3.130728497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:05.197944296 +0000 UTC m=+470.235049229" watchObservedRunningTime="2026-11-02 14:05:06.185823776 +0000 UTC m=+471.222928705" Nov 02 14:05:06.187030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:06.186987 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64d68d6d56-76wsv" podStartSLOduration=1.660373517 podStartE2EDuration="6.18697431s" totalImagesPullingTime="4.526600793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:06.185574925 +0000 UTC m=+471.222679858" watchObservedRunningTime="2026-11-02 14:05:06.18697431 +0000 UTC m=+471.224079241" Nov 02 14:05:07.122138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:07.122106 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-z2nb2" event={"ID":"bb8c30b8-0631-48c2-8e5e-f7feae79214c","Type":"ContainerStarted","Data":"204bcc7e6f5f7d083dfefbfa6e2d5e52694ad42fdf60ec456afe0ef80b45f21a"} Nov 02 14:05:07.122591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:07.122299 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:07.123967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:07.123940 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" event={"ID":"426ccdb4-cca3-4641-8cc1-af9fffbea7f3","Type":"ContainerStarted","Data":"e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738"} Nov 02 14:05:07.124083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:07.124015 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:07.168254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:07.168196 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-z2nb2" podStartSLOduration=2.235003801 podStartE2EDuration="7.168179418s" totalImagesPullingTime="4.933175617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:07.167300705 +0000 UTC m=+472.204405638" watchObservedRunningTime="2026-11-02 14:05:07.168179418 +0000 UTC m=+472.205284351" Nov 02 14:05:07.169214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:07.169172 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" podStartSLOduration=3.113421761 podStartE2EDuration="7.169157452s" totalImagesPullingTime="4.055735691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:06.237902634 +0000 UTC m=+471.275007565" watchObservedRunningTime="2026-11-02 14:05:07.169157452 +0000 UTC m=+472.206262387" Nov 02 14:05:07.214201 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:07.214145 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" podStartSLOduration=2.100272105 podStartE2EDuration="6.214132417s" totalImagesPullingTime="4.113860312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:07.212884667 +0000 UTC m=+472.249989600" watchObservedRunningTime="2026-11-02 14:05:07.214132417 +0000 UTC m=+472.251237349" Nov 02 14:05:16.121384 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:16.121351 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-4kw4d" Nov 02 14:05:17.126510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:17.126476 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:18.130479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.130452 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:05:18.130912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.130525 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-z2nb2" Nov 02 14:05:18.250118 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.250092 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d68d6d56-76wsv"] Nov 02 14:05:18.250300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.250264 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64d68d6d56-76wsv" podUID="30bc48be-afdc-4155-abd0-d81f94772f74" containerName="manager" containerID="cri-o://0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096" gracePeriod=30 Nov 02 14:05:18.496579 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.496555 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:18.617601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.617569 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-metrics-certs\") pod \"30bc48be-afdc-4155-abd0-d81f94772f74\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " Nov 02 14:05:18.617740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.617625 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bc48be-afdc-4155-abd0-d81f94772f74-kube-api-access-tqhf6\" (UniqueName: \"kubernetes.io/projected/30bc48be-afdc-4155-abd0-d81f94772f74-kube-api-access-tqhf6\") pod \"30bc48be-afdc-4155-abd0-d81f94772f74\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " Nov 02 14:05:18.617740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.617717 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-webhook-cert\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-webhook-cert\") pod \"30bc48be-afdc-4155-abd0-d81f94772f74\" (UID: \"30bc48be-afdc-4155-abd0-d81f94772f74\") " Nov 02 14:05:18.619742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.619717 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-metrics-certs" pod "30bc48be-afdc-4155-abd0-d81f94772f74" (UID: "30bc48be-afdc-4155-abd0-d81f94772f74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:05:18.619865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.619756 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bc48be-afdc-4155-abd0-d81f94772f74-kube-api-access-tqhf6" pod "30bc48be-afdc-4155-abd0-d81f94772f74" (UID: "30bc48be-afdc-4155-abd0-d81f94772f74"). InnerVolumeSpecName "kube-api-access-tqhf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:05:18.619865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.619752 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-webhook-cert" pod "30bc48be-afdc-4155-abd0-d81f94772f74" (UID: "30bc48be-afdc-4155-abd0-d81f94772f74"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:05:18.719036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.718957 4431 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-webhook-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:05:18.719036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.718986 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30bc48be-afdc-4155-abd0-d81f94772f74-metrics-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:05:18.719036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:18.718995 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhf6\" (UniqueName: \"kubernetes.io/projected/30bc48be-afdc-4155-abd0-d81f94772f74-kube-api-access-tqhf6\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:05:19.176689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.176656 4431 generic.go:356] "Generic (PLEG): container finished" podID="30bc48be-afdc-4155-abd0-d81f94772f74" containerID="0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096" exitCode=0 Nov 02 14:05:19.177188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.176707 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64d68d6d56-76wsv" Nov 02 14:05:19.177188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.176741 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d68d6d56-76wsv" event={"ID":"30bc48be-afdc-4155-abd0-d81f94772f74","Type":"ContainerDied","Data":"0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096"} Nov 02 14:05:19.177188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.176817 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64d68d6d56-76wsv" event={"ID":"30bc48be-afdc-4155-abd0-d81f94772f74","Type":"ContainerDied","Data":"f8a56ede4bb10a3f34703ef7c150b59ef83d496cf207192c0b309b2d29fb9e28"} Nov 02 14:05:19.177188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.176850 4431 scope.go:122] "RemoveContainer" containerID="0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096" Nov 02 14:05:19.185111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.185089 4431 scope.go:122] "RemoveContainer" containerID="0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096" Nov 02 14:05:19.185354 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:05:19.185336 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096\": container with ID starting with 0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096 not found: ID does not exist" containerID="0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096" Nov 02 14:05:19.185419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.185366 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096"} err="failed to get container status \"0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096\": rpc error: code = NotFound desc = could not find container \"0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096\": container with ID starting with 0a014c5f2335a24e5a746935126bb7f69e17dfd0c5f61827f331bc47e5d34096 not found: ID does not exist" Nov 02 14:05:19.232235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.232211 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64d68d6d56-76wsv"] Nov 02 14:05:19.245334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:19.245313 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64d68d6d56-76wsv"] Nov 02 14:05:21.074623 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:21.074593 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bc48be-afdc-4155-abd0-d81f94772f74" path="/var/lib/kubelet/pods/30bc48be-afdc-4155-abd0-d81f94772f74/volumes" Nov 02 14:05:21.281400 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:05:21.281363 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:05:36.124138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:36.124054 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7xw9r" Nov 02 14:05:37.125808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:37.125767 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:05:41.526546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:41.526516 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:05:51.285822 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:05:51.285766 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:05:56.054363 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.054325 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fttqq"] Nov 02 14:05:56.054735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.054714 4431 cpu_manager.go:414] "containerMap: removing container" podUID="30bc48be-afdc-4155-abd0-d81f94772f74" containerName="manager" Nov 02 14:05:56.054735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.054724 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bc48be-afdc-4155-abd0-d81f94772f74" containerName="manager" Nov 02 14:05:56.054839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.054803 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bc48be-afdc-4155-abd0-d81f94772f74" containerName="manager" Nov 02 14:05:56.056934 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.056918 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:56.079262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.079235 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fttqq"] Nov 02 14:05:56.228582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.228536 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05fb164b-a69a-4241-97d0-3705e3233692-cert\") pod \"llmisvc-controller-manager-6466c554d8-fttqq\" (UID: \"05fb164b-a69a-4241-97d0-3705e3233692\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:56.228809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.228693 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qw6w\" (UniqueName: \"kubernetes.io/projected/05fb164b-a69a-4241-97d0-3705e3233692-kube-api-access-6qw6w\") pod \"llmisvc-controller-manager-6466c554d8-fttqq\" (UID: \"05fb164b-a69a-4241-97d0-3705e3233692\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:56.329844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.329747 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05fb164b-a69a-4241-97d0-3705e3233692-cert\") pod \"llmisvc-controller-manager-6466c554d8-fttqq\" (UID: \"05fb164b-a69a-4241-97d0-3705e3233692\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:56.329992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.329895 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qw6w\" (UniqueName: \"kubernetes.io/projected/05fb164b-a69a-4241-97d0-3705e3233692-kube-api-access-6qw6w\") pod \"llmisvc-controller-manager-6466c554d8-fttqq\" (UID: \"05fb164b-a69a-4241-97d0-3705e3233692\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:56.332114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.332094 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05fb164b-a69a-4241-97d0-3705e3233692-cert\") pod \"llmisvc-controller-manager-6466c554d8-fttqq\" (UID: \"05fb164b-a69a-4241-97d0-3705e3233692\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:56.354841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.354814 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qw6w\" (UniqueName: \"kubernetes.io/projected/05fb164b-a69a-4241-97d0-3705e3233692-kube-api-access-6qw6w\") pod \"llmisvc-controller-manager-6466c554d8-fttqq\" (UID: \"05fb164b-a69a-4241-97d0-3705e3233692\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:56.368620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.368598 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:56.663088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:56.663063 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fttqq"] Nov 02 14:05:56.665513 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:05:56.665480 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fb164b_a69a_4241_97d0_3705e3233692.slice/crio-2a62c6f4359e97f2e4b93654d1727c042343c5261ce8c4a0f6544999c30f29ac WatchSource:0}: Error finding container 2a62c6f4359e97f2e4b93654d1727c042343c5261ce8c4a0f6544999c30f29ac: Status 404 returned error can't find the container with id 2a62c6f4359e97f2e4b93654d1727c042343c5261ce8c4a0f6544999c30f29ac Nov 02 14:05:57.348188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:57.348100 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" event={"ID":"05fb164b-a69a-4241-97d0-3705e3233692","Type":"ContainerStarted","Data":"8fa75161dcd90e3ea2e113ee953e042292f128dadd2527efd9668da55a70e30a"} Nov 02 14:05:57.348188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:57.348166 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" event={"ID":"05fb164b-a69a-4241-97d0-3705e3233692","Type":"ContainerStarted","Data":"2a62c6f4359e97f2e4b93654d1727c042343c5261ce8c4a0f6544999c30f29ac"} Nov 02 14:05:57.348666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:57.348197 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:05:57.392902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:05:57.392854 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" podStartSLOduration=1.124625042 podStartE2EDuration="1.392839774s" totalImagesPullingTime="268.214732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:57.392134486 +0000 UTC m=+522.429239419" watchObservedRunningTime="2026-11-02 14:05:57.392839774 +0000 UTC m=+522.429944708" Nov 02 14:06:21.280952 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:06:21.280918 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:06:28.355900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.355867 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fttqq" Nov 02 14:06:28.442624 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.442588 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv"] Nov 02 14:06:28.442871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.442838 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" podUID="23fc9928-3865-499b-99f0-4e5ec1811f63" containerName="manager" containerID="cri-o://93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4" gracePeriod=30 Nov 02 14:06:28.680034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.680006 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:06:28.787350 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.787318 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23fc9928-3865-499b-99f0-4e5ec1811f63-cert\" (UniqueName: \"kubernetes.io/secret/23fc9928-3865-499b-99f0-4e5ec1811f63-cert\") pod \"23fc9928-3865-499b-99f0-4e5ec1811f63\" (UID: \"23fc9928-3865-499b-99f0-4e5ec1811f63\") " Nov 02 14:06:28.787505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.787421 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fc9928-3865-499b-99f0-4e5ec1811f63-kube-api-access-lw8zl\" (UniqueName: \"kubernetes.io/projected/23fc9928-3865-499b-99f0-4e5ec1811f63-kube-api-access-lw8zl\") pod \"23fc9928-3865-499b-99f0-4e5ec1811f63\" (UID: \"23fc9928-3865-499b-99f0-4e5ec1811f63\") " Nov 02 14:06:28.789439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.789410 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23fc9928-3865-499b-99f0-4e5ec1811f63-cert" pod "23fc9928-3865-499b-99f0-4e5ec1811f63" (UID: "23fc9928-3865-499b-99f0-4e5ec1811f63"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:06:28.789598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.789461 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fc9928-3865-499b-99f0-4e5ec1811f63-kube-api-access-lw8zl" pod "23fc9928-3865-499b-99f0-4e5ec1811f63" (UID: "23fc9928-3865-499b-99f0-4e5ec1811f63"). InnerVolumeSpecName "kube-api-access-lw8zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:06:28.888953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.888922 4431 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23fc9928-3865-499b-99f0-4e5ec1811f63-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:06:28.888953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:28.888952 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8zl\" (UniqueName: \"kubernetes.io/projected/23fc9928-3865-499b-99f0-4e5ec1811f63-kube-api-access-lw8zl\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:06:29.495211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.495173 4431 generic.go:356] "Generic (PLEG): container finished" podID="23fc9928-3865-499b-99f0-4e5ec1811f63" containerID="93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4" exitCode=0 Nov 02 14:06:29.495603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.495224 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" Nov 02 14:06:29.495603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.495233 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" event={"ID":"23fc9928-3865-499b-99f0-4e5ec1811f63","Type":"ContainerDied","Data":"93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4"} Nov 02 14:06:29.495603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.495283 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv" event={"ID":"23fc9928-3865-499b-99f0-4e5ec1811f63","Type":"ContainerDied","Data":"b4e09547fafccbc0f5b9b0c4a3fb6e83e07f57cf7f58df8cae5aa78f98c8263e"} Nov 02 14:06:29.495603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.495312 4431 scope.go:122] "RemoveContainer" containerID="93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4" Nov 02 14:06:29.503662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.503642 4431 scope.go:122] "RemoveContainer" containerID="93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4" Nov 02 14:06:29.503961 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:06:29.503941 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4\": container with ID starting with 93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4 not found: ID does not exist" containerID="93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4" Nov 02 14:06:29.504042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.503975 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4"} err="failed to get container status \"93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4\": rpc error: code = NotFound desc = could not find container \"93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4\": container with ID starting with 93da18484ba0a54444148e59b8bba50f40206bfecf072d1ff56bf69df1fed2e4 not found: ID does not exist" Nov 02 14:06:29.555742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.555719 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv"] Nov 02 14:06:29.568730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:29.568710 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vqsrv"] Nov 02 14:06:31.074875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:06:31.074839 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fc9928-3865-499b-99f0-4e5ec1811f63" path="/var/lib/kubelet/pods/23fc9928-3865-499b-99f0-4e5ec1811f63/volumes" Nov 02 14:06:51.282556 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:06:51.282449 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:07:21.280579 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:07:21.280541 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:07:22.273020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:22.272989 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:07:27.211107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:27.211075 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:07:32.802410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:32.802374 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:07:37.895526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:37.895491 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:07:51.211236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.211201 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx"] Nov 02 14:07:51.211607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.211579 4431 cpu_manager.go:414] "containerMap: removing container" podUID="23fc9928-3865-499b-99f0-4e5ec1811f63" containerName="manager" Nov 02 14:07:51.211607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.211589 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fc9928-3865-499b-99f0-4e5ec1811f63" containerName="manager" Nov 02 14:07:51.211682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.211646 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fc9928-3865-499b-99f0-4e5ec1811f63" containerName="manager" Nov 02 14:07:51.214824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.214805 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.237561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.237542 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 14:07:51.245889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.245855 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4tfr8\"" Nov 02 14:07:51.248128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.248108 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx"] Nov 02 14:07:51.287367 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:07:51.287324 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:07:51.378282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378250 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.378282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378286 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.378466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378323 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.378466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378422 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.378466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378457 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.378562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378486 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.378562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378509 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.378622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378594 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8pp\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-kube-api-access-js8pp\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.378622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.378616 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152eb70-3989-4a4d-a6e7-5df967b974ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.479799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.479703 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.479799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.479746 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.479965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.479824 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8pp\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-kube-api-access-js8pp\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.479965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.479858 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152eb70-3989-4a4d-a6e7-5df967b974ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480048 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480090 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480145 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480183 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480203 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480207 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480475 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480494 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152eb70-3989-4a4d-a6e7-5df967b974ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480511 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.480740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.480722 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.482175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.482153 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.482458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.482440 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.504487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.504464 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8pp\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-kube-api-access-js8pp\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.504659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.504640 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-97tgx\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.527943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.527919 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:51.841263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.841188 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx"] Nov 02 14:07:51.844681 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:07:51.844655 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5152eb70_3989_4a4d_a6e7_5df967b974ac.slice/crio-69fde2699503798d1f4ab60bc88b8d7007a3c950a62e8739cdb023ca3811e7a2 WatchSource:0}: Error finding container 69fde2699503798d1f4ab60bc88b8d7007a3c950a62e8739cdb023ca3811e7a2: Status 404 returned error can't find the container with id 69fde2699503798d1f4ab60bc88b8d7007a3c950a62e8739cdb023ca3811e7a2 Nov 02 14:07:51.861864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:51.861832 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" event={"ID":"5152eb70-3989-4a4d-a6e7-5df967b974ac","Type":"ContainerStarted","Data":"69fde2699503798d1f4ab60bc88b8d7007a3c950a62e8739cdb023ca3811e7a2"} Nov 02 14:07:54.540702 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:54.540670 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:07:54.541020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:54.540736 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:07:54.541020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:54.540766 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:07:54.878990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:54.878905 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" event={"ID":"5152eb70-3989-4a4d-a6e7-5df967b974ac","Type":"ContainerStarted","Data":"e3dd0ce0de42a37ed1ecd91115c486fb9e0aae1bb8c03fbad6864e30559a67ca"} Nov 02 14:07:54.953229 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:54.953181 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podStartSLOduration=1.2599535020000001 podStartE2EDuration="3.953166477s" totalImagesPullingTime="2.693212975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:54.952599365 +0000 UTC m=+639.989704296" watchObservedRunningTime="2026-11-02 14:07:54.953166477 +0000 UTC m=+639.990271408" Nov 02 14:07:55.528985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:55.528949 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:07:55.530457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:55.530430 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-97tgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 14:07:55.530573 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:55.530480 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 14:07:56.528669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:56.528635 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-97tgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 14:07:56.529101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:56.528686 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 14:07:57.528402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:57.528370 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-97tgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 14:07:57.528574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:57.528419 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 14:07:58.528356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:58.528322 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-97tgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 14:07:58.528721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:58.528386 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 14:07:59.528545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:59.528511 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-97tgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 14:07:59.528938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:07:59.528565 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 14:08:00.528371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:00.528335 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-97tgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 14:08:00.528532 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:00.528390 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 14:08:01.529022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:01.528981 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-97tgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 02 14:08:01.529535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:01.529054 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 02 14:08:02.710398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:02.710362 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:08:02.711143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:02.711125 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:08:02.711715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:02.711693 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:08:06.296951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.296911 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc"] Nov 02 14:08:06.300559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.300533 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.306213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.306183 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.306318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.306229 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1927399f-2905-4061-8966-8a5d2d98cbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.306318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.306256 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.306578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.306555 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1927399f-2905-4061-8966-8a5d2d98cbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.307170 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.307145 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.307262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.307184 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd686\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-kube-api-access-wd686\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.307326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.307273 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.307326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.307315 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.307450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.307335 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.326017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.325984 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc"] Nov 02 14:08:06.408364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408336 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408374 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408391 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408446 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408465 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1927399f-2905-4061-8966-8a5d2d98cbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408486 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408790 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408513 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1927399f-2905-4061-8966-8a5d2d98cbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408790 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408555 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408790 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408581 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd686\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-kube-api-access-wd686\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408832 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408901 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.408977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.408958 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.409162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.409145 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.409320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.409299 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1927399f-2905-4061-8966-8a5d2d98cbda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.410791 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.410757 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.411002 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.410983 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1927399f-2905-4061-8966-8a5d2d98cbda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.433413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.433391 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd686\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-kube-api-access-wd686\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.433491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.433429 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-pxflc\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.615349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.615300 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:06.939864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.939175 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc"] Nov 02 14:08:06.940897 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:08:06.940870 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1927399f_2905_4061_8966_8a5d2d98cbda.slice/crio-ac6b21282a778fb0a310cc1d48badd537d83e3699b5135870b7b3fbc3c957c23 WatchSource:0}: Error finding container ac6b21282a778fb0a310cc1d48badd537d83e3699b5135870b7b3fbc3c957c23: Status 404 returned error can't find the container with id ac6b21282a778fb0a310cc1d48badd537d83e3699b5135870b7b3fbc3c957c23 Nov 02 14:08:06.943806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.943750 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:08:06.943902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.943837 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:08:06.943902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.943871 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:08:06.949316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:06.949287 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" event={"ID":"1927399f-2905-4061-8966-8a5d2d98cbda","Type":"ContainerStarted","Data":"ac6b21282a778fb0a310cc1d48badd537d83e3699b5135870b7b3fbc3c957c23"} Nov 02 14:08:07.955770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:07.955737 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" event={"ID":"1927399f-2905-4061-8966-8a5d2d98cbda","Type":"ContainerStarted","Data":"d1e3a8568c79cff14bb00cbd2d2c2c243f785f218e9f97ef861111dfe8d280ab"} Nov 02 14:08:08.054275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:08.054228 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podStartSLOduration=2.054215227 podStartE2EDuration="2.054215227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:08.041703932 +0000 UTC m=+653.078808866" watchObservedRunningTime="2026-11-02 14:08:08.054215227 +0000 UTC m=+653.091320158" Nov 02 14:08:08.616087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:08.616007 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:08.617562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:08.617536 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 14:08:08.617700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:08.617582 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 14:08:09.615955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:09.615923 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 14:08:09.616353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:09.615973 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 14:08:10.616354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:10.616313 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 14:08:10.616745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:10.616375 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 14:08:11.616059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:11.616019 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 14:08:11.616254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:11.616078 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 14:08:12.616595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:12.616557 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 14:08:12.617082 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:12.616611 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 14:08:13.616072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:13.616036 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 14:08:13.616240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:13.616092 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 14:08:15.045230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:15.045196 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:15.045677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:15.045605 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:15.046391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:15.046367 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:08:21.281369 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:08:21.281333 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:08:21.326492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.326462 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l"] Nov 02 14:08:21.330389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.330352 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.362343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.362317 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l"] Nov 02 14:08:21.431919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.431886 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.432080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.431932 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0881c94-9060-442e-b479-17013ddeb3ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.432080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.431955 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7d7\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-kube-api-access-vd7d7\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.432080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.432039 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.432257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.432180 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.432257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.432243 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.432373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.432315 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0881c94-9060-442e-b479-17013ddeb3ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.432373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.432350 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.432488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.432468 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533520 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7d7\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-kube-api-access-vd7d7\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533552 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533580 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533610 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533726 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0881c94-9060-442e-b479-17013ddeb3ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533799 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533849 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533878 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.533965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.533930 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0881c94-9060-442e-b479-17013ddeb3ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.534268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.534031 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.534268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.534182 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.534268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.534198 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.534412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.534380 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.534565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.534548 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0881c94-9060-442e-b479-17013ddeb3ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.536060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.536036 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.536226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.536211 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0881c94-9060-442e-b479-17013ddeb3ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.558210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.558183 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.558461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.558442 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7d7\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-kube-api-access-vd7d7\") pod \"maas-default-gateway-openshift-default-55fbd65575-jbt2l\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.643114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.643087 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:21.957138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.957066 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l"] Nov 02 14:08:21.959504 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:08:21.959477 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0881c94_9060_442e_b479_17013ddeb3ca.slice/crio-f494ad6fccc0d3c8a05db67e4cebd660d17b549652c5550ece0ca92090c22a28 WatchSource:0}: Error finding container f494ad6fccc0d3c8a05db67e4cebd660d17b549652c5550ece0ca92090c22a28: Status 404 returned error can't find the container with id f494ad6fccc0d3c8a05db67e4cebd660d17b549652c5550ece0ca92090c22a28 Nov 02 14:08:21.961563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.961529 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:08:21.961660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.961598 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:08:21.961660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:21.961627 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:08:22.027772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:22.027747 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" event={"ID":"e0881c94-9060-442e-b479-17013ddeb3ca","Type":"ContainerStarted","Data":"a41e333a41a2841c798878e3432d12c4619efe4934e743a23c5e49b9ca2f5e41"} Nov 02 14:08:22.027910 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:22.027840 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" event={"ID":"e0881c94-9060-442e-b479-17013ddeb3ca","Type":"ContainerStarted","Data":"f494ad6fccc0d3c8a05db67e4cebd660d17b549652c5550ece0ca92090c22a28"} Nov 02 14:08:22.078060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:22.077986 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podStartSLOduration=1.077968999 podStartE2EDuration="1.077968999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:22.077370446 +0000 UTC m=+667.114475381" watchObservedRunningTime="2026-11-02 14:08:22.077968999 +0000 UTC m=+667.115073935" Nov 02 14:08:22.643568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:22.643527 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:23.644670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:23.644632 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 14:08:23.645149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:23.644693 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 14:08:24.644218 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:24.644175 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 14:08:24.644424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:24.644232 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 14:08:25.644124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:25.644091 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 14:08:25.644599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:25.644142 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 14:08:26.643454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:26.643416 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 14:08:26.643661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:26.643474 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 14:08:27.643757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:27.643726 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 14:08:27.644161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:27.643792 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 14:08:28.643438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:28.643403 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 02 14:08:28.643620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:28.643465 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 02 14:08:30.074658 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:30.074623 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:30.075154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:30.075066 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:30.075867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:08:30.075846 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:08:51.281552 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:08:51.281516 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:09:02.623740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:02.623699 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:09:07.849141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:07.849103 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:09:12.194453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:12.194422 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:09:21.282418 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:09:21.282378 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:09:21.503917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:21.503890 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:09:27.354587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.354549 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q"] Nov 02 14:09:27.358729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.358708 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.380123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.380095 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q"] Nov 02 14:09:27.391062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.391035 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-555d1e\"/\"openshift-service-ca.crt\"" Nov 02 14:09:27.391208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.391070 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-555d1e\"/\"kube-root-ca.crt\"" Nov 02 14:09:27.391208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.391169 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-555d1e\"/\"default-dockercfg-9dn26\"" Nov 02 14:09:27.391314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.391233 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-555d1e\"/\"simulator-w0-555d1e-kserve-self-signed-certs\"" Nov 02 14:09:27.403038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.403020 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-model-cache\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.403125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.403061 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tls-certs\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.403125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.403083 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4v2l\" (UniqueName: \"kubernetes.io/projected/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-kube-api-access-m4v2l\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.403240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.403223 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tmp-dir\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.403291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.403278 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-home\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.403356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.403333 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-dshm\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504102 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504071 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-home\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504116 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-dshm\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504164 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-model-cache\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504222 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tls-certs\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504393 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504250 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4v2l\" (UniqueName: \"kubernetes.io/projected/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-kube-api-access-m4v2l\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504504 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-home\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504507 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tmp-dir\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504668 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504632 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-model-cache\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.504946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.504924 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tmp-dir\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.506276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.506255 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-dshm\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.506992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.506974 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tls-certs\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.530535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.530501 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4v2l\" (UniqueName: \"kubernetes.io/projected/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-kube-api-access-m4v2l\") pod \"simulator-w0-555d1e-kserve-5848449944-xqz6q\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.674183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.674154 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:27.973559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.973536 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q"] Nov 02 14:09:27.976507 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:09:27.976479 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0236687_f8bf_4b9e_a5fb_cee272e4d86e.slice/crio-28192a34d2dc26c97f828cb4a3b641644cadc3bc1e00168240ec237c05472caa WatchSource:0}: Error finding container 28192a34d2dc26c97f828cb4a3b641644cadc3bc1e00168240ec237c05472caa: Status 404 returned error can't find the container with id 28192a34d2dc26c97f828cb4a3b641644cadc3bc1e00168240ec237c05472caa Nov 02 14:09:27.978418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:27.978398 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:09:28.376643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:28.373307 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" event={"ID":"c0236687-f8bf-4b9e-a5fb-cee272e4d86e","Type":"ContainerStarted","Data":"28192a34d2dc26c97f828cb4a3b641644cadc3bc1e00168240ec237c05472caa"} Nov 02 14:09:29.380015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:29.379919 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" event={"ID":"c0236687-f8bf-4b9e-a5fb-cee272e4d86e","Type":"ContainerStarted","Data":"aff6593b02f19045c9f525ff184c812324717e0b80b0346aa89d30f242a1a863"} Nov 02 14:09:29.434539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:29.434452 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podStartSLOduration=1.34906121 podStartE2EDuration="2.434437759s" totalImagesPullingTime="1.085376549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:29.43434398 +0000 UTC m=+734.471448913" watchObservedRunningTime="2026-11-02 14:09:29.434437759 +0000 UTC m=+734.471542693" Nov 02 14:09:29.674750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:29.674706 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:31.004459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:31.004431 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:09:37.674399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:37.674364 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:37.688831 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:37.688800 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:38.417229 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:38.417195 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:09:38.432536 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:38.432509 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:09:43.820183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.820143 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj"] Nov 02 14:09:43.826478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.826457 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:43.844680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.844653 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj"] Nov 02 14:09:43.855870 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.855849 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"kube-root-ca.crt\"" Nov 02 14:09:43.855870 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.855857 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"openshift-service-ca.crt\"" Nov 02 14:09:43.891074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.891049 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"premium-w4-c0c28f-kserve-self-signed-certs\"" Nov 02 14:09:43.891074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.891065 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"default-dockercfg-hfg2t\"" Nov 02 14:09:43.958841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.958803 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-tmp-dir\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:43.959010 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.958860 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-dshm\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:43.959010 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.958957 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lfp\" (UniqueName: \"kubernetes.io/projected/e43d8539-443f-42d1-b09f-7ec8714c1137-kube-api-access-l8lfp\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:43.959110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.959040 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43d8539-443f-42d1-b09f-7ec8714c1137-tls-certs\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:43.959110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.959086 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-model-cache\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:43.959248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:43.959228 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-home\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.060527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.060491 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-home\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.060723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.060557 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-tmp-dir\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.060723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.060586 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-dshm\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.060723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.060618 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lfp\" (UniqueName: \"kubernetes.io/projected/e43d8539-443f-42d1-b09f-7ec8714c1137-kube-api-access-l8lfp\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.060723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.060682 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43d8539-443f-42d1-b09f-7ec8714c1137-tls-certs\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.061075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.060726 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-model-cache\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.061075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.060995 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-home\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.061168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.061076 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-tmp-dir\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.061168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.061106 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-model-cache\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.062988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.062962 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-dshm\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.063288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.063136 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43d8539-443f-42d1-b09f-7ec8714c1137-tls-certs\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.086207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.086150 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lfp\" (UniqueName: \"kubernetes.io/projected/e43d8539-443f-42d1-b09f-7ec8714c1137-kube-api-access-l8lfp\") pod \"premium-w4-c0c28f-kserve-7b54844d86-wg5fj\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.139353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.139329 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:44.454965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:44.454940 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj"] Nov 02 14:09:44.457654 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:09:44.457617 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode43d8539_443f_42d1_b09f_7ec8714c1137.slice/crio-69fbdcf72ebd9355de5122a5250e00d6604ae644f319f07485b4bf429039ecd5 WatchSource:0}: Error finding container 69fbdcf72ebd9355de5122a5250e00d6604ae644f319f07485b4bf429039ecd5: Status 404 returned error can't find the container with id 69fbdcf72ebd9355de5122a5250e00d6604ae644f319f07485b4bf429039ecd5 Nov 02 14:09:45.465089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:45.465057 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" event={"ID":"e43d8539-443f-42d1-b09f-7ec8714c1137","Type":"ContainerStarted","Data":"c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3"} Nov 02 14:09:45.465554 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:45.465097 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" event={"ID":"e43d8539-443f-42d1-b09f-7ec8714c1137","Type":"ContainerStarted","Data":"69fbdcf72ebd9355de5122a5250e00d6604ae644f319f07485b4bf429039ecd5"} Nov 02 14:09:45.510103 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:45.510058 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podStartSLOduration=2.510045061 podStartE2EDuration="2.510045061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:45.509130675 +0000 UTC m=+750.546235606" watchObservedRunningTime="2026-11-02 14:09:45.510045061 +0000 UTC m=+750.547149992" Nov 02 14:09:46.139862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:46.139827 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:47.928771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:47.928741 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:09:51.282097 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:09:51.282063 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:09:54.139768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:54.139735 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:54.152107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:54.152080 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:09:54.525365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:09:54.525293 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:10:00.750839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:00.750745 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:10:03.984572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:03.984535 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk"] Nov 02 14:10:03.988217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:03.988193 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.020400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.020359 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"default-dockercfg-kwgz8\"" Nov 02 14:10:04.020576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.020419 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"openshift-service-ca.crt\"" Nov 02 14:10:04.020576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.020471 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"premium-w2-4be645-kserve-self-signed-certs\"" Nov 02 14:10:04.020576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.020532 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"kube-root-ca.crt\"" Nov 02 14:10:04.022598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.022576 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk"] Nov 02 14:10:04.050171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.050144 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-model-cache\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.050309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.050184 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-dshm\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.050358 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.050309 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tls-certs\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.050358 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.050336 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4xj\" (UniqueName: \"kubernetes.io/projected/b47e4c06-0b1c-464b-b0e7-768229c9cc35-kube-api-access-tw4xj\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.050437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.050405 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-home\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.050492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.050435 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tmp-dir\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.151954 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.151901 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-model-cache\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.151954 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.151957 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-dshm\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.152212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.151991 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tls-certs\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.152212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.152019 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4xj\" (UniqueName: \"kubernetes.io/projected/b47e4c06-0b1c-464b-b0e7-768229c9cc35-kube-api-access-tw4xj\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.152212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.152073 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-home\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.152212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.152116 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tmp-dir\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.152434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.152343 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-model-cache\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.152526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.152504 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tmp-dir\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.152717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.152694 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-home\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.154721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.154696 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-dshm\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.154826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.154701 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tls-certs\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.179419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.179385 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4xj\" (UniqueName: \"kubernetes.io/projected/b47e4c06-0b1c-464b-b0e7-768229c9cc35-kube-api-access-tw4xj\") pod \"premium-w2-4be645-kserve-8d7b895db-5jjmk\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.302707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.302602 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:04.609601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:04.609576 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk"] Nov 02 14:10:04.611956 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:10:04.611930 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb47e4c06_0b1c_464b_b0e7_768229c9cc35.slice/crio-e52958cce927a465b557fc907b4e9f9e05a8d79f3f9773bcfe2de583a6c37826 WatchSource:0}: Error finding container e52958cce927a465b557fc907b4e9f9e05a8d79f3f9773bcfe2de583a6c37826: Status 404 returned error can't find the container with id e52958cce927a465b557fc907b4e9f9e05a8d79f3f9773bcfe2de583a6c37826 Nov 02 14:10:05.567439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:05.567408 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" event={"ID":"b47e4c06-0b1c-464b-b0e7-768229c9cc35","Type":"ContainerStarted","Data":"75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965"} Nov 02 14:10:05.567862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:05.567447 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" event={"ID":"b47e4c06-0b1c-464b-b0e7-768229c9cc35","Type":"ContainerStarted","Data":"e52958cce927a465b557fc907b4e9f9e05a8d79f3f9773bcfe2de583a6c37826"} Nov 02 14:10:05.612292 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:05.612243 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podStartSLOduration=2.612225038 podStartE2EDuration="2.612225038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:05.612136441 +0000 UTC m=+770.649241373" watchObservedRunningTime="2026-11-02 14:10:05.612225038 +0000 UTC m=+770.649329952" Nov 02 14:10:06.302964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:06.302928 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:07.467289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.467257 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj"] Nov 02 14:10:07.470888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.470869 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.489390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.489371 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c0c28f-openshift-default-dockercfg-z9fnh\"" Nov 02 14:10:07.504765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.504739 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj"] Nov 02 14:10:07.586397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586369 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.586509 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586415 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.586555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586504 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.586596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586578 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.586626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586608 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.586691 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586677 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.586747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586733 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.586797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586769 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.586838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.586824 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gnr\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-kube-api-access-x4gnr\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.687538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687507 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.687664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687573 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.687664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687620 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.687664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687651 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.687876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687675 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gnr\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-kube-api-access-x4gnr\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.687876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687751 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.687876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687815 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.688029 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687875 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.688029 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.687923 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.688156 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.688131 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.688274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.688248 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.688439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.688401 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.688575 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.688512 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.688736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.688716 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.689910 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.689887 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.690012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.689997 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.712539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.712517 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gnr\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-kube-api-access-x4gnr\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.712638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.712622 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:07.784920 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:07.784819 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:08.121869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:08.121786 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj"] Nov 02 14:10:08.128123 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:10:08.128092 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5edc2bd_c7dd_40b0_9e46_db8efd1428b4.slice/crio-7c5deea707e2e216c454f8fcf5d28472919d6f22ae584fca8e92b01e60b4f3ef WatchSource:0}: Error finding container 7c5deea707e2e216c454f8fcf5d28472919d6f22ae584fca8e92b01e60b4f3ef: Status 404 returned error can't find the container with id 7c5deea707e2e216c454f8fcf5d28472919d6f22ae584fca8e92b01e60b4f3ef Nov 02 14:10:08.130916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:08.130877 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:10:08.130991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:08.130965 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:10:08.131035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:08.131011 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:10:08.590677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:08.590614 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" event={"ID":"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4","Type":"ContainerStarted","Data":"f8cba83e45358de3705b49f107e2b7fae4ac0ab8a95adea159bcfa806bfdbef2"} Nov 02 14:10:08.591310 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:08.590694 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" event={"ID":"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4","Type":"ContainerStarted","Data":"7c5deea707e2e216c454f8fcf5d28472919d6f22ae584fca8e92b01e60b4f3ef"} Nov 02 14:10:08.656154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:08.656093 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" podStartSLOduration=1.656073915 podStartE2EDuration="1.656073915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:08.654995795 +0000 UTC m=+773.692100727" watchObservedRunningTime="2026-11-02 14:10:08.656073915 +0000 UTC m=+773.693178850" Nov 02 14:10:08.788907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:08.788866 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:09.793904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:09.793870 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 14:10:09.794479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:09.794447 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 14:10:10.788801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:10.788121 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 14:10:10.788801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:10.788187 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 14:10:11.788187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:11.788154 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 14:10:11.788752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:11.788721 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 14:10:12.789172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:12.789132 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 14:10:12.789802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:12.789745 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 14:10:13.792361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:13.791907 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 14:10:13.792361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:13.791967 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 14:10:14.303087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:14.303053 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:14.315736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:14.315714 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:14.634116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:14.634041 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:10:14.790080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:14.790046 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:14.790335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:14.790316 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:14.790859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:14.790843 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:10:14.828460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:14.828432 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:10:21.284410 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:10:21.284376 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:10:34.332215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:34.332174 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:10:37.933166 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:37.933123 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f"] Nov 02 14:10:37.936963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:37.936934 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:37.973526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:37.973497 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f"] Nov 02 14:10:37.980749 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:37.980715 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:10:37.981008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:37.980985 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:10:38.082061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.082030 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlv2w\" (UniqueName: \"kubernetes.io/projected/6171bd98-da10-45cf-9b91-7ec7cbe1622d-kube-api-access-qlv2w\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.082209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.082075 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.082252 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.082206 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.183293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.183221 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.183470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.183341 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.183470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.183445 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlv2w\" (UniqueName: \"kubernetes.io/projected/6171bd98-da10-45cf-9b91-7ec7cbe1622d-kube-api-access-qlv2w\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.184349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.184324 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.185735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.185712 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.208036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.208013 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlv2w\" (UniqueName: \"kubernetes.io/projected/6171bd98-da10-45cf-9b91-7ec7cbe1622d-kube-api-access-qlv2w\") pod \"maas-api-revoke-sub-e4f6e542-wbr6f\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.251859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.251832 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:38.551967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.551945 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f"] Nov 02 14:10:38.554463 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:10:38.554431 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6171bd98_da10_45cf_9b91_7ec7cbe1622d.slice/crio-cdb0a2aa7e6b0a2186512116bac928e47cc842e3f4300a3b4f1088de804de3d9 WatchSource:0}: Error finding container cdb0a2aa7e6b0a2186512116bac928e47cc842e3f4300a3b4f1088de804de3d9: Status 404 returned error can't find the container with id cdb0a2aa7e6b0a2186512116bac928e47cc842e3f4300a3b4f1088de804de3d9 Nov 02 14:10:38.762674 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:38.762639 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" event={"ID":"6171bd98-da10-45cf-9b91-7ec7cbe1622d","Type":"ContainerStarted","Data":"cdb0a2aa7e6b0a2186512116bac928e47cc842e3f4300a3b4f1088de804de3d9"} Nov 02 14:10:41.781884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:41.781846 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" event={"ID":"6171bd98-da10-45cf-9b91-7ec7cbe1622d","Type":"ContainerStarted","Data":"20530a7c908ad7943e48477eea749c93cba292a8a774bb3e8e4d12bfa5d21d7c"} Nov 02 14:10:41.826518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:41.826472 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" podStartSLOduration=2.396776129 podStartE2EDuration="4.826459578s" totalImagesPullingTime="2.429683449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:41.825310273 +0000 UTC m=+806.862415205" watchObservedRunningTime="2026-11-02 14:10:41.826459578 +0000 UTC m=+806.863564509" Nov 02 14:10:42.788209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:42.788170 4431 generic.go:356] "Generic (PLEG): container finished" podID="6171bd98-da10-45cf-9b91-7ec7cbe1622d" containerID="20530a7c908ad7943e48477eea749c93cba292a8a774bb3e8e4d12bfa5d21d7c" exitCode=0 Nov 02 14:10:42.788584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:42.788212 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" event={"ID":"6171bd98-da10-45cf-9b91-7ec7cbe1622d","Type":"ContainerDied","Data":"20530a7c908ad7943e48477eea749c93cba292a8a774bb3e8e4d12bfa5d21d7c"} Nov 02 14:10:43.940957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:43.940931 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:44.038095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.038065 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-openshift-ca\") pod \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " Nov 02 14:10:44.038220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.038122 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6171bd98-da10-45cf-9b91-7ec7cbe1622d-kube-api-access-qlv2w\" (UniqueName: \"kubernetes.io/projected/6171bd98-da10-45cf-9b91-7ec7cbe1622d-kube-api-access-qlv2w\") pod \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " Nov 02 14:10:44.038271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.038221 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-xks-ca\") pod \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\" (UID: \"6171bd98-da10-45cf-9b91-7ec7cbe1622d\") " Nov 02 14:10:44.038485 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.038464 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-openshift-ca" pod "6171bd98-da10-45cf-9b91-7ec7cbe1622d" (UID: "6171bd98-da10-45cf-9b91-7ec7cbe1622d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:44.040346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.040325 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-xks-ca" pod "6171bd98-da10-45cf-9b91-7ec7cbe1622d" (UID: "6171bd98-da10-45cf-9b91-7ec7cbe1622d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:10:44.040435 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.040339 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6171bd98-da10-45cf-9b91-7ec7cbe1622d-kube-api-access-qlv2w" pod "6171bd98-da10-45cf-9b91-7ec7cbe1622d" (UID: "6171bd98-da10-45cf-9b91-7ec7cbe1622d"). InnerVolumeSpecName "kube-api-access-qlv2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:44.139022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.138988 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:44.139022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.139017 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6171bd98-da10-45cf-9b91-7ec7cbe1622d-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:44.139183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.139030 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlv2w\" (UniqueName: \"kubernetes.io/projected/6171bd98-da10-45cf-9b91-7ec7cbe1622d-kube-api-access-qlv2w\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:44.798680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.798650 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" Nov 02 14:10:44.798887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.798677 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f" event={"ID":"6171bd98-da10-45cf-9b91-7ec7cbe1622d","Type":"ContainerDied","Data":"cdb0a2aa7e6b0a2186512116bac928e47cc842e3f4300a3b4f1088de804de3d9"} Nov 02 14:10:44.798887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:44.798722 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdb0a2aa7e6b0a2186512116bac928e47cc842e3f4300a3b4f1088de804de3d9" Nov 02 14:10:49.442432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.442396 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk"] Nov 02 14:10:49.442854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.442839 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6171bd98-da10-45cf-9b91-7ec7cbe1622d" containerName="revoke-keys" Nov 02 14:10:49.442854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.442854 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6171bd98-da10-45cf-9b91-7ec7cbe1622d" containerName="revoke-keys" Nov 02 14:10:49.442951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.442942 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6171bd98-da10-45cf-9b91-7ec7cbe1622d" containerName="revoke-keys" Nov 02 14:10:49.446491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.446458 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.466204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.466171 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk"] Nov 02 14:10:49.476957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.476934 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:10:49.477055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.476969 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:10:49.489158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.489130 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbz28\" (UniqueName: \"kubernetes.io/projected/06224c7c-dd28-461c-9615-3ed655fab3f6-kube-api-access-kbz28\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.489261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.489185 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.489339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.489320 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.590572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.590546 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbz28\" (UniqueName: \"kubernetes.io/projected/06224c7c-dd28-461c-9615-3ed655fab3f6-kube-api-access-kbz28\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.590703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.590584 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.590703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.590663 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.591260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.591235 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.593058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.593030 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.614728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.614702 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbz28\" (UniqueName: \"kubernetes.io/projected/06224c7c-dd28-461c-9615-3ed655fab3f6-kube-api-access-kbz28\") pod \"maas-api-revoke-sub-b108d0d7-9h6lk\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:49.758842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:49.758763 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:50.054764 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:50.054738 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk"] Nov 02 14:10:50.057529 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:10:50.057489 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06224c7c_dd28_461c_9615_3ed655fab3f6.slice/crio-b393840ee19f2e921acefe221198eea5491a46c42a13026e49c5e13c3b96b5ed WatchSource:0}: Error finding container b393840ee19f2e921acefe221198eea5491a46c42a13026e49c5e13c3b96b5ed: Status 404 returned error can't find the container with id b393840ee19f2e921acefe221198eea5491a46c42a13026e49c5e13c3b96b5ed Nov 02 14:10:50.836356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:50.836325 4431 generic.go:356] "Generic (PLEG): container finished" podID="06224c7c-dd28-461c-9615-3ed655fab3f6" containerID="be72b20c94f13234c5c7131284916169fdc5684e6675fa646e631dda27333863" exitCode=0 Nov 02 14:10:50.836727 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:50.836370 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" event={"ID":"06224c7c-dd28-461c-9615-3ed655fab3f6","Type":"ContainerDied","Data":"be72b20c94f13234c5c7131284916169fdc5684e6675fa646e631dda27333863"} Nov 02 14:10:50.836727 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:50.836449 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" event={"ID":"06224c7c-dd28-461c-9615-3ed655fab3f6","Type":"ContainerStarted","Data":"b393840ee19f2e921acefe221198eea5491a46c42a13026e49c5e13c3b96b5ed"} Nov 02 14:10:51.224360 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:51.224329 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:10:51.281565 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:10:51.281535 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:10:51.988347 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:51.988324 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:52.014669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.014646 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-openshift-ca\") pod \"06224c7c-dd28-461c-9615-3ed655fab3f6\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " Nov 02 14:10:52.014807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.014688 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06224c7c-dd28-461c-9615-3ed655fab3f6-kube-api-access-kbz28\" (UniqueName: \"kubernetes.io/projected/06224c7c-dd28-461c-9615-3ed655fab3f6-kube-api-access-kbz28\") pod \"06224c7c-dd28-461c-9615-3ed655fab3f6\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " Nov 02 14:10:52.014892 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.014874 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-xks-ca\") pod \"06224c7c-dd28-461c-9615-3ed655fab3f6\" (UID: \"06224c7c-dd28-461c-9615-3ed655fab3f6\") " Nov 02 14:10:52.015101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.015075 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-openshift-ca" pod "06224c7c-dd28-461c-9615-3ed655fab3f6" (UID: "06224c7c-dd28-461c-9615-3ed655fab3f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:52.015286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.015269 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:52.016825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.016749 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06224c7c-dd28-461c-9615-3ed655fab3f6-kube-api-access-kbz28" pod "06224c7c-dd28-461c-9615-3ed655fab3f6" (UID: "06224c7c-dd28-461c-9615-3ed655fab3f6"). InnerVolumeSpecName "kube-api-access-kbz28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:52.016825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.016754 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-xks-ca" pod "06224c7c-dd28-461c-9615-3ed655fab3f6" (UID: "06224c7c-dd28-461c-9615-3ed655fab3f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:10:52.116482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.116389 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbz28\" (UniqueName: \"kubernetes.io/projected/06224c7c-dd28-461c-9615-3ed655fab3f6-kube-api-access-kbz28\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:52.116482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.116430 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06224c7c-dd28-461c-9615-3ed655fab3f6-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:52.850165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.850131 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" event={"ID":"06224c7c-dd28-461c-9615-3ed655fab3f6","Type":"ContainerDied","Data":"b393840ee19f2e921acefe221198eea5491a46c42a13026e49c5e13c3b96b5ed"} Nov 02 14:10:52.850165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.850160 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk" Nov 02 14:10:52.850374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:52.850179 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b393840ee19f2e921acefe221198eea5491a46c42a13026e49c5e13c3b96b5ed" Nov 02 14:10:53.370357 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.370327 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2"] Nov 02 14:10:53.370750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.370737 4431 cpu_manager.go:414] "containerMap: removing container" podUID="06224c7c-dd28-461c-9615-3ed655fab3f6" containerName="revoke-keys" Nov 02 14:10:53.370836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.370750 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06224c7c-dd28-461c-9615-3ed655fab3f6" containerName="revoke-keys" Nov 02 14:10:53.370836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.370833 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="06224c7c-dd28-461c-9615-3ed655fab3f6" containerName="revoke-keys" Nov 02 14:10:53.374162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.374145 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.405312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.405290 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:10:53.405312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.405297 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:10:53.407321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.407303 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2"] Nov 02 14:10:53.429180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.429155 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.429278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.429197 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479gs\" (UniqueName: \"kubernetes.io/projected/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-kube-api-access-479gs\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.429327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.429307 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.530079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.530054 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.530185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.530100 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479gs\" (UniqueName: \"kubernetes.io/projected/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-kube-api-access-479gs\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.530185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.530172 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.530850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.530832 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.532661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.532628 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.557023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.556997 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-479gs\" (UniqueName: \"kubernetes.io/projected/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-kube-api-access-479gs\") pod \"maas-api-revoke-sub-addaa4b1-8k4h2\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:53.687985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:53.687956 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:54.002041 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:54.002011 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2"] Nov 02 14:10:54.860982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:54.860949 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" event={"ID":"34495bd4-f8d3-4c29-9b14-2d63f112b6a0","Type":"ContainerStarted","Data":"77907e04e78c2e2661b4b2eccfcdca505e10613fb57fecc82c5f9731d10e52ed"} Nov 02 14:10:54.861377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:54.860998 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" event={"ID":"34495bd4-f8d3-4c29-9b14-2d63f112b6a0","Type":"ContainerStarted","Data":"a04ce20a69a8cc67a3eed6c8d8250ea009e5e44da0cd24f4b55e48e99b572651"} Nov 02 14:10:54.907257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:54.907214 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" podStartSLOduration=1.907201216 podStartE2EDuration="1.907201216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:54.904288611 +0000 UTC m=+819.941393543" watchObservedRunningTime="2026-11-02 14:10:54.907201216 +0000 UTC m=+819.944306149" Nov 02 14:10:55.867276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:55.867238 4431 generic.go:356] "Generic (PLEG): container finished" podID="34495bd4-f8d3-4c29-9b14-2d63f112b6a0" containerID="77907e04e78c2e2661b4b2eccfcdca505e10613fb57fecc82c5f9731d10e52ed" exitCode=0 Nov 02 14:10:55.867648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:55.867314 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" event={"ID":"34495bd4-f8d3-4c29-9b14-2d63f112b6a0","Type":"ContainerDied","Data":"77907e04e78c2e2661b4b2eccfcdca505e10613fb57fecc82c5f9731d10e52ed"} Nov 02 14:10:56.471179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.471134 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw"] Nov 02 14:10:56.474939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.474914 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.503899 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.503878 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-758617\"/\"test-model-758617-kserve-self-signed-certs\"" Nov 02 14:10:56.504206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.504192 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-758617\"/\"default-dockercfg-gplvs\"" Nov 02 14:10:56.504266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.504213 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-758617\"/\"openshift-service-ca.crt\"" Nov 02 14:10:56.504365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.504350 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-758617\"/\"kube-root-ca.crt\"" Nov 02 14:10:56.506418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.506401 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw"] Nov 02 14:10:56.557641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.557617 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-home\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.557751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.557644 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-model-cache\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.557751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.557673 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsmwq\" (UniqueName: \"kubernetes.io/projected/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-kube-api-access-xsmwq\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.557907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.557893 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tmp-dir\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.557958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.557941 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-dshm\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.558030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.558009 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tls-certs\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.658673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.658636 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-home\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.658673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.658672 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-model-cache\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.658916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.658700 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsmwq\" (UniqueName: \"kubernetes.io/projected/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-kube-api-access-xsmwq\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.658916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.658836 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tmp-dir\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.658916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.658906 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-dshm\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.659083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.658930 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tls-certs\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.659143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.659093 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-model-cache\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.659143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.659109 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-home\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.659251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.659206 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tmp-dir\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.661261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.661237 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-dshm\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.661383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.661366 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tls-certs\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.683597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.683567 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsmwq\" (UniqueName: \"kubernetes.io/projected/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-kube-api-access-xsmwq\") pod \"test-model-758617-kserve-5ff99b9f99-rk6tw\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.788023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.787934 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:10:56.993244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:56.993215 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:57.062687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.062617 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-xks-ca\") pod \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " Nov 02 14:10:57.062836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.062800 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-kube-api-access-479gs\" (UniqueName: \"kubernetes.io/projected/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-kube-api-access-479gs\") pod \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " Nov 02 14:10:57.062883 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.062834 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-openshift-ca\") pod \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\" (UID: \"34495bd4-f8d3-4c29-9b14-2d63f112b6a0\") " Nov 02 14:10:57.063289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.063254 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-openshift-ca" pod "34495bd4-f8d3-4c29-9b14-2d63f112b6a0" (UID: "34495bd4-f8d3-4c29-9b14-2d63f112b6a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:10:57.064724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.064698 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-xks-ca" pod "34495bd4-f8d3-4c29-9b14-2d63f112b6a0" (UID: "34495bd4-f8d3-4c29-9b14-2d63f112b6a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:10:57.064859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.064842 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-kube-api-access-479gs" pod "34495bd4-f8d3-4c29-9b14-2d63f112b6a0" (UID: "34495bd4-f8d3-4c29-9b14-2d63f112b6a0"). InnerVolumeSpecName "kube-api-access-479gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:10:57.090053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.090034 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw"] Nov 02 14:10:57.091762 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:10:57.091741 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbda82a6_a948_4d2f_a49e_bc0ca92b4fb9.slice/crio-eba63562d72ec5ce6b48c06bc549e1b8d6b270896e2b760e3c3e99fe7ea5c3f4 WatchSource:0}: Error finding container eba63562d72ec5ce6b48c06bc549e1b8d6b270896e2b760e3c3e99fe7ea5c3f4: Status 404 returned error can't find the container with id eba63562d72ec5ce6b48c06bc549e1b8d6b270896e2b760e3c3e99fe7ea5c3f4 Nov 02 14:10:57.164390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.164210 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479gs\" (UniqueName: \"kubernetes.io/projected/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-kube-api-access-479gs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:57.164390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.164241 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:57.164390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.164255 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34495bd4-f8d3-4c29-9b14-2d63f112b6a0-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:10:57.878527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.878494 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" Nov 02 14:10:57.878527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.878510 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2" event={"ID":"34495bd4-f8d3-4c29-9b14-2d63f112b6a0","Type":"ContainerDied","Data":"a04ce20a69a8cc67a3eed6c8d8250ea009e5e44da0cd24f4b55e48e99b572651"} Nov 02 14:10:57.878804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.878555 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a04ce20a69a8cc67a3eed6c8d8250ea009e5e44da0cd24f4b55e48e99b572651" Nov 02 14:10:57.880069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.880041 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" event={"ID":"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9","Type":"ContainerStarted","Data":"165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf"} Nov 02 14:10:57.880210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.880078 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" event={"ID":"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9","Type":"ContainerStarted","Data":"eba63562d72ec5ce6b48c06bc549e1b8d6b270896e2b760e3c3e99fe7ea5c3f4"} Nov 02 14:10:57.933496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:57.933402 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" podStartSLOduration=1.9333853250000002 podStartE2EDuration="1.933385325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:57.929232658 +0000 UTC m=+822.966337601" watchObservedRunningTime="2026-11-02 14:10:57.933385325 +0000 UTC m=+822.970490260" Nov 02 14:10:58.788030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:10:58.788005 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:11:06.369794 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.369746 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l"] Nov 02 14:11:06.370217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.370201 4431 cpu_manager.go:414] "containerMap: removing container" podUID="34495bd4-f8d3-4c29-9b14-2d63f112b6a0" containerName="revoke-keys" Nov 02 14:11:06.370256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.370217 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34495bd4-f8d3-4c29-9b14-2d63f112b6a0" containerName="revoke-keys" Nov 02 14:11:06.370288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.370283 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="34495bd4-f8d3-4c29-9b14-2d63f112b6a0" containerName="revoke-keys" Nov 02 14:11:06.373330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.373312 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.394800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.394749 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l"] Nov 02 14:11:06.405429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.405405 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:11:06.405526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.405449 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:11:06.447503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.447474 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.447628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.447513 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtnpj\" (UniqueName: \"kubernetes.io/projected/924bd9ed-d7d8-439b-a946-416c3296a4e2-kube-api-access-xtnpj\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.447683 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.447668 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.549119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.549094 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.549231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.549150 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.549231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.549174 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtnpj\" (UniqueName: \"kubernetes.io/projected/924bd9ed-d7d8-439b-a946-416c3296a4e2-kube-api-access-xtnpj\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.549998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.549974 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.551564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.551540 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.576381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.576362 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtnpj\" (UniqueName: \"kubernetes.io/projected/924bd9ed-d7d8-439b-a946-416c3296a4e2-kube-api-access-xtnpj\") pod \"maas-api-revoke-sub-2b2b76a9-hs46l\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.686762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.686743 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:06.788819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.788792 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:11:06.801809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.801763 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:11:06.936300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.936275 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:11:06.986421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:06.986397 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l"] Nov 02 14:11:06.988556 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:06.988530 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924bd9ed_d7d8_439b_a946_416c3296a4e2.slice/crio-931f348110bb8c1165623b27f824de07eebadbd6d8271983b3cd5c1a303c8c91 WatchSource:0}: Error finding container 931f348110bb8c1165623b27f824de07eebadbd6d8271983b3cd5c1a303c8c91: Status 404 returned error can't find the container with id 931f348110bb8c1165623b27f824de07eebadbd6d8271983b3cd5c1a303c8c91 Nov 02 14:11:07.789535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.789508 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq"] Nov 02 14:11:07.793177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.793152 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.811636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.811614 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-555d1e-openshift-default-dockercfg-s5d5d\"" Nov 02 14:11:07.827077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.827050 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq"] Nov 02 14:11:07.861372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861332 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgd8\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-kube-api-access-ffgd8\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.861527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861402 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.861589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861517 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.861589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861562 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df52e7-8bdd-444d-8942-0df2310259cd-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.861704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861640 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.861704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861672 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df52e7-8bdd-444d-8942-0df2310259cd-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.861821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861762 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.861873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861848 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.861923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.861895 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.932616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.932582 4431 generic.go:356] "Generic (PLEG): container finished" podID="924bd9ed-d7d8-439b-a946-416c3296a4e2" containerID="6f435e57ddd07e7f6a17fbd4af0a544653b7efb54b02344fd8a2e2089d703033" exitCode=0 Nov 02 14:11:07.932812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.932664 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" event={"ID":"924bd9ed-d7d8-439b-a946-416c3296a4e2","Type":"ContainerDied","Data":"6f435e57ddd07e7f6a17fbd4af0a544653b7efb54b02344fd8a2e2089d703033"} Nov 02 14:11:07.932812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.932726 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" event={"ID":"924bd9ed-d7d8-439b-a946-416c3296a4e2","Type":"ContainerStarted","Data":"931f348110bb8c1165623b27f824de07eebadbd6d8271983b3cd5c1a303c8c91"} Nov 02 14:11:07.963255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963219 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgd8\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-kube-api-access-ffgd8\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963280 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963338 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963370 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df52e7-8bdd-444d-8942-0df2310259cd-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963407 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963429 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df52e7-8bdd-444d-8942-0df2310259cd-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963475 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963502 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963543 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963821 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.963980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.963920 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.964278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.964124 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.964388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.964367 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.964651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.964594 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df52e7-8bdd-444d-8942-0df2310259cd-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.966559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.966539 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df52e7-8bdd-444d-8942-0df2310259cd-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.967391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.967365 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.992171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.992144 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgd8\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-kube-api-access-ffgd8\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:07.992337 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:07.992173 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:08.106504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:08.106415 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:08.447143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:08.447120 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq"] Nov 02 14:11:08.448824 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:08.448798 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10df52e7_8bdd_444d_8942_0df2310259cd.slice/crio-a3f9ca44ea86b23a0d173f424d7ec2347fa440e1ab4c7de99807f41f00de452b WatchSource:0}: Error finding container a3f9ca44ea86b23a0d173f424d7ec2347fa440e1ab4c7de99807f41f00de452b: Status 404 returned error can't find the container with id a3f9ca44ea86b23a0d173f424d7ec2347fa440e1ab4c7de99807f41f00de452b Nov 02 14:11:08.451066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:08.451031 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:08.451135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:08.451097 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:08.451135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:08.451128 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:08.940264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:08.940223 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" event={"ID":"10df52e7-8bdd-444d-8942-0df2310259cd","Type":"ContainerStarted","Data":"19702f4521c1064ea290389feeb0ea22b3456c0c2db8e27584e3d3c032cbb409"} Nov 02 14:11:08.940652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:08.940288 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" event={"ID":"10df52e7-8bdd-444d-8942-0df2310259cd","Type":"ContainerStarted","Data":"a3f9ca44ea86b23a0d173f424d7ec2347fa440e1ab4c7de99807f41f00de452b"} Nov 02 14:11:09.002066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:08.998464 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn"] Nov 02 14:11:09.036396 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.035392 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.039882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.039832 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" podStartSLOduration=2.039817829 podStartE2EDuration="2.039817829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:09.036225994 +0000 UTC m=+834.073331027" watchObservedRunningTime="2026-11-02 14:11:09.039817829 +0000 UTC m=+834.076922761" Nov 02 14:11:09.042075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.042051 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn"] Nov 02 14:11:09.076415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.075443 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.076415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.075493 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.076415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.075559 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2cl\" (UniqueName: \"kubernetes.io/projected/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-kube-api-access-zt2cl\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.107042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.106939 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:09.110478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.110447 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 14:11:09.110709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.110500 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 14:11:09.131317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.131295 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:09.183445 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.183408 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-xks-ca\") pod \"924bd9ed-d7d8-439b-a946-416c3296a4e2\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " Nov 02 14:11:09.183857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.183835 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924bd9ed-d7d8-439b-a946-416c3296a4e2-kube-api-access-xtnpj\" (UniqueName: \"kubernetes.io/projected/924bd9ed-d7d8-439b-a946-416c3296a4e2-kube-api-access-xtnpj\") pod \"924bd9ed-d7d8-439b-a946-416c3296a4e2\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " Nov 02 14:11:09.184044 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.184018 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-openshift-ca\") pod \"924bd9ed-d7d8-439b-a946-416c3296a4e2\" (UID: \"924bd9ed-d7d8-439b-a946-416c3296a4e2\") " Nov 02 14:11:09.185582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.185554 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2cl\" (UniqueName: \"kubernetes.io/projected/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-kube-api-access-zt2cl\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.186078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.186057 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.187552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.187092 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.188301 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.187757 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.189571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.189548 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-openshift-ca" pod "924bd9ed-d7d8-439b-a946-416c3296a4e2" (UID: "924bd9ed-d7d8-439b-a946-416c3296a4e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:09.199264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.199170 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-xks-ca" pod "924bd9ed-d7d8-439b-a946-416c3296a4e2" (UID: "924bd9ed-d7d8-439b-a946-416c3296a4e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:09.201678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.201643 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924bd9ed-d7d8-439b-a946-416c3296a4e2-kube-api-access-xtnpj" pod "924bd9ed-d7d8-439b-a946-416c3296a4e2" (UID: "924bd9ed-d7d8-439b-a946-416c3296a4e2"). InnerVolumeSpecName "kube-api-access-xtnpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:09.202394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.202340 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.219644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.219616 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2cl\" (UniqueName: \"kubernetes.io/projected/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-kube-api-access-zt2cl\") pod \"maas-api-revoke-sub-16ce1ca2-8smrn\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.292592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.292500 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:09.292592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.292548 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtnpj\" (UniqueName: \"kubernetes.io/projected/924bd9ed-d7d8-439b-a946-416c3296a4e2-kube-api-access-xtnpj\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:09.292592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.292564 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/924bd9ed-d7d8-439b-a946-416c3296a4e2-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:09.351524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.351494 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:09.682803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.681343 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn"] Nov 02 14:11:09.947692 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.947607 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" event={"ID":"5c02f8fc-dc0b-469f-bad2-a0c382126ac1","Type":"ContainerStarted","Data":"5bf0f607572c7018141aeba329b349a3af3589058983923bf8793f30ba8caa45"} Nov 02 14:11:09.949209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.949185 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" Nov 02 14:11:09.949338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.949182 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l" event={"ID":"924bd9ed-d7d8-439b-a946-416c3296a4e2","Type":"ContainerDied","Data":"931f348110bb8c1165623b27f824de07eebadbd6d8271983b3cd5c1a303c8c91"} Nov 02 14:11:09.949338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:09.949320 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931f348110bb8c1165623b27f824de07eebadbd6d8271983b3cd5c1a303c8c91" Nov 02 14:11:10.107370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:10.107337 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 14:11:10.107525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:10.107392 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 14:11:10.851169 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:10.851138 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:11:10.956833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:10.956796 4431 generic.go:356] "Generic (PLEG): container finished" podID="5c02f8fc-dc0b-469f-bad2-a0c382126ac1" containerID="7a79231295263092df4db19c86543759340902e6fe524aaaae940675c0d2e50a" exitCode=0 Nov 02 14:11:10.957245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:10.956879 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" event={"ID":"5c02f8fc-dc0b-469f-bad2-a0c382126ac1","Type":"ContainerDied","Data":"7a79231295263092df4db19c86543759340902e6fe524aaaae940675c0d2e50a"} Nov 02 14:11:11.107929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.107849 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 14:11:11.107929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.107900 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 14:11:11.622809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.620924 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8"] Nov 02 14:11:11.622809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.621735 4431 cpu_manager.go:414] "containerMap: removing container" podUID="924bd9ed-d7d8-439b-a946-416c3296a4e2" containerName="revoke-keys" Nov 02 14:11:11.622809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.621755 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924bd9ed-d7d8-439b-a946-416c3296a4e2" containerName="revoke-keys" Nov 02 14:11:11.622809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.621932 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="924bd9ed-d7d8-439b-a946-416c3296a4e2" containerName="revoke-keys" Nov 02 14:11:11.636465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.636440 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.649046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.649023 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8"] Nov 02 14:11:11.659837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.659452 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c0c28f\"/\"distinct-2-w4-c0c28f-kserve-self-signed-certs\"" Nov 02 14:11:11.721366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.721320 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28d6443-555d-4b37-b286-ccd55a957b01-tls-certs\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.721543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.721407 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hw6r\" (UniqueName: \"kubernetes.io/projected/d28d6443-555d-4b37-b286-ccd55a957b01-kube-api-access-9hw6r\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.721543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.721437 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-model-cache\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.721543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.721459 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-tmp-dir\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.721543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.721524 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-home\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.721752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.721552 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-dshm\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.822421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.822372 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28d6443-555d-4b37-b286-ccd55a957b01-tls-certs\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.822621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.822448 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hw6r\" (UniqueName: \"kubernetes.io/projected/d28d6443-555d-4b37-b286-ccd55a957b01-kube-api-access-9hw6r\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.822621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.822469 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-model-cache\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.822621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.822507 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-tmp-dir\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.822621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.822537 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-home\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.822621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.822558 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-dshm\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.822965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.822922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-model-cache\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.823024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.822968 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-tmp-dir\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.823082 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.823051 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-home\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.825114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.825084 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-dshm\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.825335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.825317 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28d6443-555d-4b37-b286-ccd55a957b01-tls-certs\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.873845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.870206 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hw6r\" (UniqueName: \"kubernetes.io/projected/d28d6443-555d-4b37-b286-ccd55a957b01-kube-api-access-9hw6r\") pod \"distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:11.952793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:11.952748 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:12.108017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.107985 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 14:11:12.108587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.108044 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 14:11:12.198039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.198007 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:12.299500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.299472 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8"] Nov 02 14:11:12.300609 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:12.300584 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd28d6443_555d_4b37_b286_ccd55a957b01.slice/crio-a9cb77eb3337dfd8686d65e95551858a985b88eefc3c33c53e012cac36c1bd51 WatchSource:0}: Error finding container a9cb77eb3337dfd8686d65e95551858a985b88eefc3c33c53e012cac36c1bd51: Status 404 returned error can't find the container with id a9cb77eb3337dfd8686d65e95551858a985b88eefc3c33c53e012cac36c1bd51 Nov 02 14:11:12.328611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.328586 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-kube-api-access-zt2cl\" (UniqueName: \"kubernetes.io/projected/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-kube-api-access-zt2cl\") pod \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " Nov 02 14:11:12.328700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.328639 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-xks-ca\") pod \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " Nov 02 14:11:12.328700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.328682 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-openshift-ca\") pod \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\" (UID: \"5c02f8fc-dc0b-469f-bad2-a0c382126ac1\") " Nov 02 14:11:12.329141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.329114 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-openshift-ca" pod "5c02f8fc-dc0b-469f-bad2-a0c382126ac1" (UID: "5c02f8fc-dc0b-469f-bad2-a0c382126ac1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:12.330574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.330547 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-kube-api-access-zt2cl" pod "5c02f8fc-dc0b-469f-bad2-a0c382126ac1" (UID: "5c02f8fc-dc0b-469f-bad2-a0c382126ac1"). InnerVolumeSpecName "kube-api-access-zt2cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:12.330677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.330629 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-xks-ca" pod "5c02f8fc-dc0b-469f-bad2-a0c382126ac1" (UID: "5c02f8fc-dc0b-469f-bad2-a0c382126ac1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:12.430171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.430086 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt2cl\" (UniqueName: \"kubernetes.io/projected/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-kube-api-access-zt2cl\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:12.430171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.430118 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:12.430171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.430131 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f8fc-dc0b-469f-bad2-a0c382126ac1-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:12.997511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.997383 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" Nov 02 14:11:12.997814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.997382 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn" event={"ID":"5c02f8fc-dc0b-469f-bad2-a0c382126ac1","Type":"ContainerDied","Data":"5bf0f607572c7018141aeba329b349a3af3589058983923bf8793f30ba8caa45"} Nov 02 14:11:12.997942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:12.997825 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf0f607572c7018141aeba329b349a3af3589058983923bf8793f30ba8caa45" Nov 02 14:11:13.000285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:13.000254 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" event={"ID":"d28d6443-555d-4b37-b286-ccd55a957b01","Type":"ContainerStarted","Data":"3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0"} Nov 02 14:11:13.000410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:13.000297 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" event={"ID":"d28d6443-555d-4b37-b286-ccd55a957b01","Type":"ContainerStarted","Data":"a9cb77eb3337dfd8686d65e95551858a985b88eefc3c33c53e012cac36c1bd51"} Nov 02 14:11:13.046500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:13.046451 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podStartSLOduration=2.046437455 podStartE2EDuration="2.046437455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:13.045244421 +0000 UTC m=+838.082349353" watchObservedRunningTime="2026-11-02 14:11:13.046437455 +0000 UTC m=+838.083542406" Nov 02 14:11:13.107925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:13.107851 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 14:11:13.107925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:13.107900 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 14:11:13.953575 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:13.953539 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:14.107237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:14.107207 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 14:11:14.107424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:14.107259 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 14:11:15.880255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:15.880220 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:15.880820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:15.880798 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:15.881453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:15.881435 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:11:19.480697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.480667 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx"] Nov 02 14:11:19.481134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.481119 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5c02f8fc-dc0b-469f-bad2-a0c382126ac1" containerName="revoke-keys" Nov 02 14:11:19.481174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.481134 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c02f8fc-dc0b-469f-bad2-a0c382126ac1" containerName="revoke-keys" Nov 02 14:11:19.481211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.481203 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c02f8fc-dc0b-469f-bad2-a0c382126ac1" containerName="revoke-keys" Nov 02 14:11:19.485543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.485523 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.516111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.516087 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:11:19.516227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.516087 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:11:19.518600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.518580 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx"] Nov 02 14:11:19.603467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.603438 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.603617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.603478 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.603686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.603633 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6x4\" (UniqueName: \"kubernetes.io/projected/fd91cb61-2547-4361-bb8e-1bebae482cac-kube-api-access-rp6x4\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.705049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.705020 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.705203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.705058 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.705203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.705147 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6x4\" (UniqueName: \"kubernetes.io/projected/fd91cb61-2547-4361-bb8e-1bebae482cac-kube-api-access-rp6x4\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.705969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.705945 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.707482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.707461 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.738280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.738222 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6x4\" (UniqueName: \"kubernetes.io/projected/fd91cb61-2547-4361-bb8e-1bebae482cac-kube-api-access-rp6x4\") pod \"maas-api-revoke-sub-08845dc3-kj8mx\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:19.799364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:19.799340 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:20.107847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:20.107822 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx"] Nov 02 14:11:20.109406 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:20.109384 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd91cb61_2547_4361_bb8e_1bebae482cac.slice/crio-59272e5cee68ae6082db8bac600b8545249b840d014aaa52f653b7297efe56d4 WatchSource:0}: Error finding container 59272e5cee68ae6082db8bac600b8545249b840d014aaa52f653b7297efe56d4: Status 404 returned error can't find the container with id 59272e5cee68ae6082db8bac600b8545249b840d014aaa52f653b7297efe56d4 Nov 02 14:11:21.053537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:21.053503 4431 generic.go:356] "Generic (PLEG): container finished" podID="fd91cb61-2547-4361-bb8e-1bebae482cac" containerID="dc44ae52caa41fc3ca8b3bcb7c716a02807f587acdd02dfb135bf775ea48b978" exitCode=0 Nov 02 14:11:21.054014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:21.053586 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" event={"ID":"fd91cb61-2547-4361-bb8e-1bebae482cac","Type":"ContainerDied","Data":"dc44ae52caa41fc3ca8b3bcb7c716a02807f587acdd02dfb135bf775ea48b978"} Nov 02 14:11:21.054014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:21.053650 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" event={"ID":"fd91cb61-2547-4361-bb8e-1bebae482cac","Type":"ContainerStarted","Data":"59272e5cee68ae6082db8bac600b8545249b840d014aaa52f653b7297efe56d4"} Nov 02 14:11:21.288046 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:11:21.287961 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:11:21.953342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:21.953305 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:21.965769 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:21.965747 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:22.074648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.074615 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:11:22.226705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.226674 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:22.333248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.333212 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-openshift-ca\") pod \"fd91cb61-2547-4361-bb8e-1bebae482cac\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " Nov 02 14:11:22.333441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.333416 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd91cb61-2547-4361-bb8e-1bebae482cac-kube-api-access-rp6x4\" (UniqueName: \"kubernetes.io/projected/fd91cb61-2547-4361-bb8e-1bebae482cac-kube-api-access-rp6x4\") pod \"fd91cb61-2547-4361-bb8e-1bebae482cac\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " Nov 02 14:11:22.333525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.333474 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-xks-ca\") pod \"fd91cb61-2547-4361-bb8e-1bebae482cac\" (UID: \"fd91cb61-2547-4361-bb8e-1bebae482cac\") " Nov 02 14:11:22.333650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.333625 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-openshift-ca" pod "fd91cb61-2547-4361-bb8e-1bebae482cac" (UID: "fd91cb61-2547-4361-bb8e-1bebae482cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:22.333862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.333843 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:22.335536 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.335511 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd91cb61-2547-4361-bb8e-1bebae482cac-kube-api-access-rp6x4" pod "fd91cb61-2547-4361-bb8e-1bebae482cac" (UID: "fd91cb61-2547-4361-bb8e-1bebae482cac"). InnerVolumeSpecName "kube-api-access-rp6x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:22.335660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.335618 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-xks-ca" pod "fd91cb61-2547-4361-bb8e-1bebae482cac" (UID: "fd91cb61-2547-4361-bb8e-1bebae482cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:22.434458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.434436 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp6x4\" (UniqueName: \"kubernetes.io/projected/fd91cb61-2547-4361-bb8e-1bebae482cac-kube-api-access-rp6x4\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:22.434458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:22.434457 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd91cb61-2547-4361-bb8e-1bebae482cac-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:23.065165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:23.065134 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" Nov 02 14:11:23.065325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:23.065186 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx" event={"ID":"fd91cb61-2547-4361-bb8e-1bebae482cac","Type":"ContainerDied","Data":"59272e5cee68ae6082db8bac600b8545249b840d014aaa52f653b7297efe56d4"} Nov 02 14:11:23.065325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:23.065238 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59272e5cee68ae6082db8bac600b8545249b840d014aaa52f653b7297efe56d4" Nov 02 14:11:30.152258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.152185 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l"] Nov 02 14:11:30.152653 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.152639 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fd91cb61-2547-4361-bb8e-1bebae482cac" containerName="revoke-keys" Nov 02 14:11:30.152711 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.152656 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd91cb61-2547-4361-bb8e-1bebae482cac" containerName="revoke-keys" Nov 02 14:11:30.152747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.152732 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd91cb61-2547-4361-bb8e-1bebae482cac" containerName="revoke-keys" Nov 02 14:11:30.156109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.156094 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.172710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.172685 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:11:30.185761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.185738 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:11:30.188054 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.188034 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l"] Nov 02 14:11:30.308667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.308648 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.308799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.308691 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjc8n\" (UniqueName: \"kubernetes.io/projected/e239d438-e725-48ee-93f0-a41c6163722a-kube-api-access-hjc8n\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.308799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.308765 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.409898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.409842 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjc8n\" (UniqueName: \"kubernetes.io/projected/e239d438-e725-48ee-93f0-a41c6163722a-kube-api-access-hjc8n\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.409898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.409885 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.410020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.409963 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.410579 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.410551 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.412294 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.412274 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.434331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.434304 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjc8n\" (UniqueName: \"kubernetes.io/projected/e239d438-e725-48ee-93f0-a41c6163722a-kube-api-access-hjc8n\") pod \"maas-api-revoke-sub-44150d36-pzv9l\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.469126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.469105 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:30.769847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:30.769814 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l"] Nov 02 14:11:30.772404 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:30.772372 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode239d438_e725_48ee_93f0_a41c6163722a.slice/crio-d8f4937237e74d7a2a6588e17c1e09d794b5edbc7cf97ea319407feb9866395b WatchSource:0}: Error finding container d8f4937237e74d7a2a6588e17c1e09d794b5edbc7cf97ea319407feb9866395b: Status 404 returned error can't find the container with id d8f4937237e74d7a2a6588e17c1e09d794b5edbc7cf97ea319407feb9866395b Nov 02 14:11:31.105437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:31.105407 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" event={"ID":"e239d438-e725-48ee-93f0-a41c6163722a","Type":"ContainerStarted","Data":"a62e510b161590e7c5a81617ab231f29ce63ad69856ab796db5b8d6b41649373"} Nov 02 14:11:31.105560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:31.105461 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" event={"ID":"e239d438-e725-48ee-93f0-a41c6163722a","Type":"ContainerStarted","Data":"d8f4937237e74d7a2a6588e17c1e09d794b5edbc7cf97ea319407feb9866395b"} Nov 02 14:11:31.165372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:31.165329 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" podStartSLOduration=1.16531515 podStartE2EDuration="1.16531515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:31.164546455 +0000 UTC m=+856.201651400" watchObservedRunningTime="2026-11-02 14:11:31.16531515 +0000 UTC m=+856.202420082" Nov 02 14:11:32.111711 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.111677 4431 generic.go:356] "Generic (PLEG): container finished" podID="e239d438-e725-48ee-93f0-a41c6163722a" containerID="a62e510b161590e7c5a81617ab231f29ce63ad69856ab796db5b8d6b41649373" exitCode=0 Nov 02 14:11:32.111907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.111745 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" event={"ID":"e239d438-e725-48ee-93f0-a41c6163722a","Type":"ContainerDied","Data":"a62e510b161590e7c5a81617ab231f29ce63ad69856ab796db5b8d6b41649373"} Nov 02 14:11:32.260717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.260687 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk"] Nov 02 14:11:32.264059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.264044 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.285204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.285176 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk"] Nov 02 14:11:32.337814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.337760 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.337960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.337887 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42rv\" (UniqueName: \"kubernetes.io/projected/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-kube-api-access-v42rv\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.338015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.337986 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.438602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.438567 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.438753 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.438723 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42rv\" (UniqueName: \"kubernetes.io/projected/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-kube-api-access-v42rv\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.438868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.438852 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.439231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.439199 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.441128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.441105 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.463859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.463834 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42rv\" (UniqueName: \"kubernetes.io/projected/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-kube-api-access-v42rv\") pod \"maas-api-revoke-sub-74eed019-bpvnk\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.576608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.576581 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:32.875802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:32.875755 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk"] Nov 02 14:11:32.877694 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:32.877665 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f595db_ea5d_4d04_a03e_b05a65dd5ce5.slice/crio-f62efb36cc2f0aa492eccf363da5b060eb9ead3fad3bc6e1439e02196efaea19 WatchSource:0}: Error finding container f62efb36cc2f0aa492eccf363da5b060eb9ead3fad3bc6e1439e02196efaea19: Status 404 returned error can't find the container with id f62efb36cc2f0aa492eccf363da5b060eb9ead3fad3bc6e1439e02196efaea19 Nov 02 14:11:33.118596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.118556 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" event={"ID":"47f595db-ea5d-4d04-a03e-b05a65dd5ce5","Type":"ContainerStarted","Data":"f62efb36cc2f0aa492eccf363da5b060eb9ead3fad3bc6e1439e02196efaea19"} Nov 02 14:11:33.249301 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.249279 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:33.347889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.347857 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e239d438-e725-48ee-93f0-a41c6163722a-kube-api-access-hjc8n\" (UniqueName: \"kubernetes.io/projected/e239d438-e725-48ee-93f0-a41c6163722a-kube-api-access-hjc8n\") pod \"e239d438-e725-48ee-93f0-a41c6163722a\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " Nov 02 14:11:33.348351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.347919 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-xks-ca\") pod \"e239d438-e725-48ee-93f0-a41c6163722a\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " Nov 02 14:11:33.348351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.347991 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-openshift-ca\") pod \"e239d438-e725-48ee-93f0-a41c6163722a\" (UID: \"e239d438-e725-48ee-93f0-a41c6163722a\") " Nov 02 14:11:33.348351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.348309 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-openshift-ca" pod "e239d438-e725-48ee-93f0-a41c6163722a" (UID: "e239d438-e725-48ee-93f0-a41c6163722a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:33.348477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.348399 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:33.350018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.349995 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e239d438-e725-48ee-93f0-a41c6163722a-kube-api-access-hjc8n" pod "e239d438-e725-48ee-93f0-a41c6163722a" (UID: "e239d438-e725-48ee-93f0-a41c6163722a"). InnerVolumeSpecName "kube-api-access-hjc8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:33.350109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.350028 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-xks-ca" pod "e239d438-e725-48ee-93f0-a41c6163722a" (UID: "e239d438-e725-48ee-93f0-a41c6163722a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:33.449153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.449128 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjc8n\" (UniqueName: \"kubernetes.io/projected/e239d438-e725-48ee-93f0-a41c6163722a-kube-api-access-hjc8n\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:33.449225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.449159 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e239d438-e725-48ee-93f0-a41c6163722a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:33.591469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:33.591442 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:11:34.125792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:34.125695 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" Nov 02 14:11:34.125792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:34.125729 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l" event={"ID":"e239d438-e725-48ee-93f0-a41c6163722a","Type":"ContainerDied","Data":"d8f4937237e74d7a2a6588e17c1e09d794b5edbc7cf97ea319407feb9866395b"} Nov 02 14:11:34.126030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:34.125808 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f4937237e74d7a2a6588e17c1e09d794b5edbc7cf97ea319407feb9866395b" Nov 02 14:11:34.127554 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:34.127524 4431 generic.go:356] "Generic (PLEG): container finished" podID="47f595db-ea5d-4d04-a03e-b05a65dd5ce5" containerID="3374394f037ebb5becb59dadaffe31a63d3264302fa850d0ada215853e430bf6" exitCode=0 Nov 02 14:11:34.127703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:34.127553 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" event={"ID":"47f595db-ea5d-4d04-a03e-b05a65dd5ce5","Type":"ContainerDied","Data":"3374394f037ebb5becb59dadaffe31a63d3264302fa850d0ada215853e430bf6"} Nov 02 14:11:35.279981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.279957 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:35.366155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.366128 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-kube-api-access-v42rv\" (UniqueName: \"kubernetes.io/projected/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-kube-api-access-v42rv\") pod \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " Nov 02 14:11:35.366300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.366167 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-xks-ca\") pod \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " Nov 02 14:11:35.366300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.366204 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-openshift-ca\") pod \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\" (UID: \"47f595db-ea5d-4d04-a03e-b05a65dd5ce5\") " Nov 02 14:11:35.366582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.366557 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-openshift-ca" pod "47f595db-ea5d-4d04-a03e-b05a65dd5ce5" (UID: "47f595db-ea5d-4d04-a03e-b05a65dd5ce5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:35.368402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.368371 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-kube-api-access-v42rv" pod "47f595db-ea5d-4d04-a03e-b05a65dd5ce5" (UID: "47f595db-ea5d-4d04-a03e-b05a65dd5ce5"). InnerVolumeSpecName "kube-api-access-v42rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:35.368503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.368447 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-xks-ca" pod "47f595db-ea5d-4d04-a03e-b05a65dd5ce5" (UID: "47f595db-ea5d-4d04-a03e-b05a65dd5ce5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:35.467298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.467269 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42rv\" (UniqueName: \"kubernetes.io/projected/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-kube-api-access-v42rv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:35.467298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.467296 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:35.467448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:35.467306 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f595db-ea5d-4d04-a03e-b05a65dd5ce5-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:36.140847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:36.140808 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" event={"ID":"47f595db-ea5d-4d04-a03e-b05a65dd5ce5","Type":"ContainerDied","Data":"f62efb36cc2f0aa492eccf363da5b060eb9ead3fad3bc6e1439e02196efaea19"} Nov 02 14:11:36.140993 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:36.140864 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f62efb36cc2f0aa492eccf363da5b060eb9ead3fad3bc6e1439e02196efaea19" Nov 02 14:11:36.140993 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:36.140820 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk" Nov 02 14:11:37.761998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.761969 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx"] Nov 02 14:11:37.762395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.762381 4431 cpu_manager.go:414] "containerMap: removing container" podUID="47f595db-ea5d-4d04-a03e-b05a65dd5ce5" containerName="revoke-keys" Nov 02 14:11:37.762395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.762392 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f595db-ea5d-4d04-a03e-b05a65dd5ce5" containerName="revoke-keys" Nov 02 14:11:37.762471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.762404 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e239d438-e725-48ee-93f0-a41c6163722a" containerName="revoke-keys" Nov 02 14:11:37.762471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.762409 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e239d438-e725-48ee-93f0-a41c6163722a" containerName="revoke-keys" Nov 02 14:11:37.762539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.762496 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f595db-ea5d-4d04-a03e-b05a65dd5ce5" containerName="revoke-keys" Nov 02 14:11:37.762539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.762504 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e239d438-e725-48ee-93f0-a41c6163722a" containerName="revoke-keys" Nov 02 14:11:37.765768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.765753 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.786619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.786591 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.786743 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.786636 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.786816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.786752 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.786936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.786834 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lxd\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-kube-api-access-65lxd\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.786936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.786883 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4788beeb-e260-499f-b1fe-1c5198de154e-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.787064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.786946 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.787064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.787013 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.787064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.787047 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.787207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.787071 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4788beeb-e260-499f-b1fe-1c5198de154e-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.802427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.802404 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx"] Nov 02 14:11:37.888503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888479 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lxd\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-kube-api-access-65lxd\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.888636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888526 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4788beeb-e260-499f-b1fe-1c5198de154e-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.888636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888543 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.888636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888560 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.888636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888580 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.888887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888691 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4788beeb-e260-499f-b1fe-1c5198de154e-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.888887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888753 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.888887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888808 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.889058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888929 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.889058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.888988 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.889174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.889151 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.889271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.889248 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.889330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.889315 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.889537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.889512 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4788beeb-e260-499f-b1fe-1c5198de154e-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.890814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.890796 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.891043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.891027 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4788beeb-e260-499f-b1fe-1c5198de154e-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.914031 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.914010 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:37.914434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:37.914417 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lxd\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-kube-api-access-65lxd\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:38.020681 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.020615 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5"] Nov 02 14:11:38.025792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.025759 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.041842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.041820 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4be645-openshift-default-dockercfg-2ds72\"" Nov 02 14:11:38.050680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.050656 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5"] Nov 02 14:11:38.080766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.080739 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:38.091944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.091917 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.092037 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.092017 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.092162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.092142 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nskf\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-kube-api-access-4nskf\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.092258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.092235 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.092380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.092349 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.092447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.092397 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.092585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.092558 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.092647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.092621 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.092703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.092690 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.193505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.193476 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.193635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.193525 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nskf\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-kube-api-access-4nskf\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.193635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.193564 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.193635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.193604 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.193766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.193635 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.193766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.193684 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.193766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.193716 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.193953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.193920 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.194052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.194035 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.194110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.194063 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.194165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.194108 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.194280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.194258 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.194280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.194269 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.194428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.194333 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.196330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.196312 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.196578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.196560 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.219562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.219536 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.219732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.219714 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nskf\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-kube-api-access-4nskf\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.339300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.339247 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:38.403073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.403047 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx"] Nov 02 14:11:38.407535 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:38.407509 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4788beeb_e260_499f_b1fe_1c5198de154e.slice/crio-8fd3632305f16d1d98137a7c1fca69ef3356c4333ce4c804902e6210c3b98974 WatchSource:0}: Error finding container 8fd3632305f16d1d98137a7c1fca69ef3356c4333ce4c804902e6210c3b98974: Status 404 returned error can't find the container with id 8fd3632305f16d1d98137a7c1fca69ef3356c4333ce4c804902e6210c3b98974 Nov 02 14:11:38.409717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.409686 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:38.409813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.409751 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:38.409859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.409827 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:38.649073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.649048 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5"] Nov 02 14:11:38.651227 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:38.651195 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1592382_9355_4bff_aefe_1f4e7b5e4aa8.slice/crio-db8ca96312f695e32fed568797deca466ed9842081b47f13634d072ba48db4ae WatchSource:0}: Error finding container db8ca96312f695e32fed568797deca466ed9842081b47f13634d072ba48db4ae: Status 404 returned error can't find the container with id db8ca96312f695e32fed568797deca466ed9842081b47f13634d072ba48db4ae Nov 02 14:11:38.654023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.653975 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:38.654102 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.654069 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:38.654135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:38.654118 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:39.166802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:39.164955 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" event={"ID":"4788beeb-e260-499f-b1fe-1c5198de154e","Type":"ContainerStarted","Data":"0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4"} Nov 02 14:11:39.166802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:39.165028 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" event={"ID":"4788beeb-e260-499f-b1fe-1c5198de154e","Type":"ContainerStarted","Data":"8fd3632305f16d1d98137a7c1fca69ef3356c4333ce4c804902e6210c3b98974"} Nov 02 14:11:39.168113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:39.168078 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" event={"ID":"d1592382-9355-4bff-aefe-1f4e7b5e4aa8","Type":"ContainerStarted","Data":"1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066"} Nov 02 14:11:39.168240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:39.168130 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" event={"ID":"d1592382-9355-4bff-aefe-1f4e7b5e4aa8","Type":"ContainerStarted","Data":"db8ca96312f695e32fed568797deca466ed9842081b47f13634d072ba48db4ae"} Nov 02 14:11:39.227951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:39.227885 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podStartSLOduration=2.227871012 podStartE2EDuration="2.227871012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:39.226295691 +0000 UTC m=+864.263400626" watchObservedRunningTime="2026-11-02 14:11:39.227871012 +0000 UTC m=+864.264975945" Nov 02 14:11:39.339899 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:39.339868 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:39.341202 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:39.341175 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 14:11:39.341329 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:39.341267 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 14:11:40.082001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:40.081932 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:40.090906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:40.090877 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 14:11:40.091144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:40.091115 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 14:11:40.340600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:40.340530 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 14:11:40.341016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:40.340880 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 14:11:41.086468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:41.086429 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 14:11:41.086646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:41.086486 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 14:11:41.339820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:41.339713 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 14:11:41.339820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:41.339764 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 14:11:42.006800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.006099 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podStartSLOduration=5.006078925 podStartE2EDuration="5.006078925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:39.287116709 +0000 UTC m=+864.324221652" watchObservedRunningTime="2026-11-02 14:11:42.006078925 +0000 UTC m=+867.043183858" Nov 02 14:11:42.023342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.023307 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv"] Nov 02 14:11:42.027972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.027947 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.042143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.042116 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv"] Nov 02 14:11:42.050303 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.050122 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.050303 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.050242 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.050303 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.050276 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldhn\" (UniqueName: \"kubernetes.io/projected/5d80f20f-9f84-465d-8fb6-f93ac92b2140-kube-api-access-wldhn\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.051097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.051078 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:11:42.051413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.051394 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:11:42.127198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.127160 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 14:11:42.127383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.127220 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 14:11:42.151231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.151203 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.151646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.151628 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.151747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.151734 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldhn\" (UniqueName: \"kubernetes.io/projected/5d80f20f-9f84-465d-8fb6-f93ac92b2140-kube-api-access-wldhn\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.152309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.152284 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.153984 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.153962 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.221482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.221382 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldhn\" (UniqueName: \"kubernetes.io/projected/5d80f20f-9f84-465d-8fb6-f93ac92b2140-kube-api-access-wldhn\") pod \"maas-api-revoke-sub-b5b8af3b-v77vv\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.340119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.340048 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 14:11:42.340304 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.340283 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 14:11:42.346048 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.346025 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:42.714308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:42.714272 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv"] Nov 02 14:11:43.090801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:43.086136 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 14:11:43.090801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:43.086193 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 14:11:43.239886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:43.239841 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" event={"ID":"5d80f20f-9f84-465d-8fb6-f93ac92b2140","Type":"ContainerStarted","Data":"b784deef4fc89c82da6228b36e75f082090a7279bdf4ece06849ce709fa0d5bb"} Nov 02 14:11:43.240071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:43.239944 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" event={"ID":"5d80f20f-9f84-465d-8fb6-f93ac92b2140","Type":"ContainerStarted","Data":"92bccbca882fe4c0709ebc535fb1ce5a3812d97e8a57e34f4f2ba39466100dd3"} Nov 02 14:11:43.340084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:43.340033 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 14:11:43.340349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:43.340084 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 14:11:44.093111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.093014 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 14:11:44.093111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.093077 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 14:11:44.248048 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.248017 4431 generic.go:356] "Generic (PLEG): container finished" podID="5d80f20f-9f84-465d-8fb6-f93ac92b2140" containerID="b784deef4fc89c82da6228b36e75f082090a7279bdf4ece06849ce709fa0d5bb" exitCode=0 Nov 02 14:11:44.248203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.248109 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" event={"ID":"5d80f20f-9f84-465d-8fb6-f93ac92b2140","Type":"ContainerDied","Data":"b784deef4fc89c82da6228b36e75f082090a7279bdf4ece06849ce709fa0d5bb"} Nov 02 14:11:44.340144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.340104 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 14:11:44.340301 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.340171 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 14:11:44.870010 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.869975 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk"] Nov 02 14:11:44.873871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.873856 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:44.897012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:44.896987 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk"] Nov 02 14:11:45.018151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.018108 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqrc\" (UniqueName: \"kubernetes.io/projected/53392b50-8434-40be-b5f8-cbfce8e003cd-kube-api-access-xhqrc\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.018151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.018154 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.018409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.018330 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.081258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.081222 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 14:11:45.081439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.081282 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 14:11:45.119061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.119023 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqrc\" (UniqueName: \"kubernetes.io/projected/53392b50-8434-40be-b5f8-cbfce8e003cd-kube-api-access-xhqrc\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.119061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.119067 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.119592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.119164 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.119875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.119848 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.121695 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.121632 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.144838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.144809 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqrc\" (UniqueName: \"kubernetes.io/projected/53392b50-8434-40be-b5f8-cbfce8e003cd-kube-api-access-xhqrc\") pod \"maas-api-revoke-sub-c3b20bc0-788vk\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.187462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.187429 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:45.546819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.535376 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:45.546819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.536810 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:45.556824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.551029 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:11:45.562879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.562850 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk"] Nov 02 14:11:45.601449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.601193 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:45.780340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.777222 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d80f20f-9f84-465d-8fb6-f93ac92b2140-kube-api-access-wldhn\" (UniqueName: \"kubernetes.io/projected/5d80f20f-9f84-465d-8fb6-f93ac92b2140-kube-api-access-wldhn\") pod \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " Nov 02 14:11:45.780340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.777306 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-xks-ca\") pod \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " Nov 02 14:11:45.780340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.777436 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-openshift-ca\") pod \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\" (UID: \"5d80f20f-9f84-465d-8fb6-f93ac92b2140\") " Nov 02 14:11:45.781318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.781024 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-openshift-ca" pod "5d80f20f-9f84-465d-8fb6-f93ac92b2140" (UID: "5d80f20f-9f84-465d-8fb6-f93ac92b2140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:45.789322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.789292 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d80f20f-9f84-465d-8fb6-f93ac92b2140-kube-api-access-wldhn" pod "5d80f20f-9f84-465d-8fb6-f93ac92b2140" (UID: "5d80f20f-9f84-465d-8fb6-f93ac92b2140"). InnerVolumeSpecName "kube-api-access-wldhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:45.789450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.789403 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-xks-ca" pod "5d80f20f-9f84-465d-8fb6-f93ac92b2140" (UID: "5d80f20f-9f84-465d-8fb6-f93ac92b2140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:45.879152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.879113 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wldhn\" (UniqueName: \"kubernetes.io/projected/5d80f20f-9f84-465d-8fb6-f93ac92b2140-kube-api-access-wldhn\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:45.879152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.879152 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:45.879361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.879166 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d80f20f-9f84-465d-8fb6-f93ac92b2140-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:45.992907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.992865 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn"] Nov 02 14:11:45.993608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.993584 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5d80f20f-9f84-465d-8fb6-f93ac92b2140" containerName="revoke-keys" Nov 02 14:11:45.993745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.993610 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d80f20f-9f84-465d-8fb6-f93ac92b2140" containerName="revoke-keys" Nov 02 14:11:45.993819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.993769 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d80f20f-9f84-465d-8fb6-f93ac92b2140" containerName="revoke-keys" Nov 02 14:11:45.997684 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:45.997648 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.029631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.029599 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn"] Nov 02 14:11:46.087505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.087410 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:46.088836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.088813 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:46.099668 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.099641 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:11:46.184493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.184449 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jfj\" (UniqueName: \"kubernetes.io/projected/625ce4aa-7e7e-444e-91ca-767367e2ca01-kube-api-access-98jfj\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.185141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.185119 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.185412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.185391 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.265394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.265367 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" Nov 02 14:11:46.265951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.265356 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv" event={"ID":"5d80f20f-9f84-465d-8fb6-f93ac92b2140","Type":"ContainerDied","Data":"92bccbca882fe4c0709ebc535fb1ce5a3812d97e8a57e34f4f2ba39466100dd3"} Nov 02 14:11:46.266070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.265990 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92bccbca882fe4c0709ebc535fb1ce5a3812d97e8a57e34f4f2ba39466100dd3" Nov 02 14:11:46.268935 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.268898 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" event={"ID":"53392b50-8434-40be-b5f8-cbfce8e003cd","Type":"ContainerStarted","Data":"d091fd92a7257f748a1dbb6ab27eba090fb17b88803d5225ce5c83533e83b971"} Nov 02 14:11:46.269088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.268955 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" event={"ID":"53392b50-8434-40be-b5f8-cbfce8e003cd","Type":"ContainerStarted","Data":"e12856d3d755f2cf441463cad934f78e8ae1b8cea03332ebf8e9d2a7b15c65f4"} Nov 02 14:11:46.289807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.286281 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.289807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.286418 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jfj\" (UniqueName: \"kubernetes.io/projected/625ce4aa-7e7e-444e-91ca-767367e2ca01-kube-api-access-98jfj\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.289807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.286473 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.289807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.287614 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.293179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.293116 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.332324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.326899 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jfj\" (UniqueName: \"kubernetes.io/projected/625ce4aa-7e7e-444e-91ca-767367e2ca01-kube-api-access-98jfj\") pod \"maas-api-revoke-sub-c3f56f5f-s5ctn\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:46.334466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.334407 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" podStartSLOduration=2.334375557 podStartE2EDuration="2.334375557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:46.327513399 +0000 UTC m=+871.364618332" watchObservedRunningTime="2026-11-02 14:11:46.334375557 +0000 UTC m=+871.371480491" Nov 02 14:11:46.615487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:46.615448 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:47.223211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:47.223124 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn"] Nov 02 14:11:47.282866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:47.282821 4431 generic.go:356] "Generic (PLEG): container finished" podID="53392b50-8434-40be-b5f8-cbfce8e003cd" containerID="d091fd92a7257f748a1dbb6ab27eba090fb17b88803d5225ce5c83533e83b971" exitCode=0 Nov 02 14:11:47.283579 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:47.283065 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" event={"ID":"53392b50-8434-40be-b5f8-cbfce8e003cd","Type":"ContainerDied","Data":"d091fd92a7257f748a1dbb6ab27eba090fb17b88803d5225ce5c83533e83b971"} Nov 02 14:11:47.285960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:47.285938 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" event={"ID":"625ce4aa-7e7e-444e-91ca-767367e2ca01","Type":"ContainerStarted","Data":"25e9c9e237672e25943026c9822afc81601b5b8b1a42a4355455a135bec68615"} Nov 02 14:11:48.293219 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.293186 4431 generic.go:356] "Generic (PLEG): container finished" podID="625ce4aa-7e7e-444e-91ca-767367e2ca01" containerID="6047d2c4563145c2e96d2ec5c7c97bdc74c28aad2116e677ee5cdd55ce8e180d" exitCode=0 Nov 02 14:11:48.293558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.293264 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" event={"ID":"625ce4aa-7e7e-444e-91ca-767367e2ca01","Type":"ContainerDied","Data":"6047d2c4563145c2e96d2ec5c7c97bdc74c28aad2116e677ee5cdd55ce8e180d"} Nov 02 14:11:48.441768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.441744 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:48.527082 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.527048 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-openshift-ca\") pod \"53392b50-8434-40be-b5f8-cbfce8e003cd\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " Nov 02 14:11:48.527238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.527154 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-xks-ca\") pod \"53392b50-8434-40be-b5f8-cbfce8e003cd\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " Nov 02 14:11:48.527307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.527234 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53392b50-8434-40be-b5f8-cbfce8e003cd-kube-api-access-xhqrc\" (UniqueName: \"kubernetes.io/projected/53392b50-8434-40be-b5f8-cbfce8e003cd-kube-api-access-xhqrc\") pod \"53392b50-8434-40be-b5f8-cbfce8e003cd\" (UID: \"53392b50-8434-40be-b5f8-cbfce8e003cd\") " Nov 02 14:11:48.527411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.527390 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-openshift-ca" pod "53392b50-8434-40be-b5f8-cbfce8e003cd" (UID: "53392b50-8434-40be-b5f8-cbfce8e003cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:48.527587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.527570 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:48.529235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.529213 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53392b50-8434-40be-b5f8-cbfce8e003cd-kube-api-access-xhqrc" pod "53392b50-8434-40be-b5f8-cbfce8e003cd" (UID: "53392b50-8434-40be-b5f8-cbfce8e003cd"). InnerVolumeSpecName "kube-api-access-xhqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:48.529343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.529287 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-xks-ca" pod "53392b50-8434-40be-b5f8-cbfce8e003cd" (UID: "53392b50-8434-40be-b5f8-cbfce8e003cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:48.628691 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.628624 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53392b50-8434-40be-b5f8-cbfce8e003cd-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:48.628691 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:48.628652 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhqrc\" (UniqueName: \"kubernetes.io/projected/53392b50-8434-40be-b5f8-cbfce8e003cd-kube-api-access-xhqrc\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:49.299389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.299359 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" Nov 02 14:11:49.299766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.299391 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk" event={"ID":"53392b50-8434-40be-b5f8-cbfce8e003cd","Type":"ContainerDied","Data":"e12856d3d755f2cf441463cad934f78e8ae1b8cea03332ebf8e9d2a7b15c65f4"} Nov 02 14:11:49.299766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.299432 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e12856d3d755f2cf441463cad934f78e8ae1b8cea03332ebf8e9d2a7b15c65f4" Nov 02 14:11:49.450383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.450358 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:49.539439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.539415 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-xks-ca\") pod \"625ce4aa-7e7e-444e-91ca-767367e2ca01\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " Nov 02 14:11:49.539577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.539458 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-openshift-ca\") pod \"625ce4aa-7e7e-444e-91ca-767367e2ca01\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " Nov 02 14:11:49.539577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.539547 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625ce4aa-7e7e-444e-91ca-767367e2ca01-kube-api-access-98jfj\" (UniqueName: \"kubernetes.io/projected/625ce4aa-7e7e-444e-91ca-767367e2ca01-kube-api-access-98jfj\") pod \"625ce4aa-7e7e-444e-91ca-767367e2ca01\" (UID: \"625ce4aa-7e7e-444e-91ca-767367e2ca01\") " Nov 02 14:11:49.539833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.539809 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-openshift-ca" pod "625ce4aa-7e7e-444e-91ca-767367e2ca01" (UID: "625ce4aa-7e7e-444e-91ca-767367e2ca01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:49.541391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.541368 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-xks-ca" pod "625ce4aa-7e7e-444e-91ca-767367e2ca01" (UID: "625ce4aa-7e7e-444e-91ca-767367e2ca01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:49.541506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.541490 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625ce4aa-7e7e-444e-91ca-767367e2ca01-kube-api-access-98jfj" pod "625ce4aa-7e7e-444e-91ca-767367e2ca01" (UID: "625ce4aa-7e7e-444e-91ca-767367e2ca01"). InnerVolumeSpecName "kube-api-access-98jfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:49.641351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.641328 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:49.641351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.641352 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625ce4aa-7e7e-444e-91ca-767367e2ca01-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:49.641512 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:49.641362 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jfj\" (UniqueName: \"kubernetes.io/projected/625ce4aa-7e7e-444e-91ca-767367e2ca01-kube-api-access-98jfj\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:50.305457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:50.305428 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" Nov 02 14:11:50.305841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:50.305466 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn" event={"ID":"625ce4aa-7e7e-444e-91ca-767367e2ca01","Type":"ContainerDied","Data":"25e9c9e237672e25943026c9822afc81601b5b8b1a42a4355455a135bec68615"} Nov 02 14:11:50.305841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:50.305517 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e9c9e237672e25943026c9822afc81601b5b8b1a42a4355455a135bec68615" Nov 02 14:11:51.283929 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:11:51.283819 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:11:52.928010 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.927980 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc"] Nov 02 14:11:52.928465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.928450 4431 cpu_manager.go:414] "containerMap: removing container" podUID="53392b50-8434-40be-b5f8-cbfce8e003cd" containerName="revoke-keys" Nov 02 14:11:52.928520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.928466 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53392b50-8434-40be-b5f8-cbfce8e003cd" containerName="revoke-keys" Nov 02 14:11:52.928520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.928476 4431 cpu_manager.go:414] "containerMap: removing container" podUID="625ce4aa-7e7e-444e-91ca-767367e2ca01" containerName="revoke-keys" Nov 02 14:11:52.928520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.928481 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625ce4aa-7e7e-444e-91ca-767367e2ca01" containerName="revoke-keys" Nov 02 14:11:52.928630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.928582 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="53392b50-8434-40be-b5f8-cbfce8e003cd" containerName="revoke-keys" Nov 02 14:11:52.928630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.928589 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="625ce4aa-7e7e-444e-91ca-767367e2ca01" containerName="revoke-keys" Nov 02 14:11:52.931062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.931041 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:52.964130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:52.964105 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc"] Nov 02 14:11:53.076424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076395 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.076424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076429 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.076628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076547 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.076628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076578 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.076628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076597 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.076810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076668 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.076810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076713 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.076810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076729 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnwt\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-kube-api-access-wrnwt\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.076810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.076770 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178310 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178239 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178309 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178338 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178425 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178460 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178491 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178527 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178565 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178589 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnwt\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-kube-api-access-wrnwt\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.178953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.178913 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.179268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.179068 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.179373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.179285 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.179373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.179287 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.179665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.179637 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.181119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.181098 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.181220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.181193 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.205420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.205379 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.210013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.209989 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnwt\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-kube-api-access-wrnwt\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.245100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.245076 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:53.567253 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.567223 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc"] Nov 02 14:11:53.568701 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:53.568676 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59f3098_fd86_4ed3_ae76_12e47eaf4e85.slice/crio-167454f2eb76ffd8f1b4c14d302ad7ddbdef707e4b6158f543e31579f10acb4f WatchSource:0}: Error finding container 167454f2eb76ffd8f1b4c14d302ad7ddbdef707e4b6158f543e31579f10acb4f: Status 404 returned error can't find the container with id 167454f2eb76ffd8f1b4c14d302ad7ddbdef707e4b6158f543e31579f10acb4f Nov 02 14:11:53.571065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.571032 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:53.571135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.571102 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:53.571172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:53.571135 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:11:54.330449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:54.330413 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" event={"ID":"e59f3098-fd86-4ed3-ae76-12e47eaf4e85","Type":"ContainerStarted","Data":"bd7aed879dd2f5904a5abff168082320c38dc5fe89d3c805fe5b56a3c20ebb57"} Nov 02 14:11:54.330889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:54.330472 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" event={"ID":"e59f3098-fd86-4ed3-ae76-12e47eaf4e85","Type":"ContainerStarted","Data":"167454f2eb76ffd8f1b4c14d302ad7ddbdef707e4b6158f543e31579f10acb4f"} Nov 02 14:11:54.408599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:54.408546 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" podStartSLOduration=2.4085329939999998 podStartE2EDuration="2.408532994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:54.393866422 +0000 UTC m=+879.430971356" watchObservedRunningTime="2026-11-02 14:11:54.408532994 +0000 UTC m=+879.445637926" Nov 02 14:11:55.245508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:55.245475 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:11:55.247192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:55.247163 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:11:55.247323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:55.247215 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 14:11:56.246447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.246413 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:11:56.246862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.246464 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 14:11:56.361628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.361598 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96"] Nov 02 14:11:56.366124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.366098 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.406846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.402035 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:11:56.406846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.402086 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:11:56.407174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.407143 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96"] Nov 02 14:11:56.518437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.518344 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfv2j\" (UniqueName: \"kubernetes.io/projected/30cd348d-5e87-4114-a53c-ffb9095bd1d8-kube-api-access-gfv2j\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.518437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.518433 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.518678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.518507 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.619479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.619441 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.619663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.619577 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfv2j\" (UniqueName: \"kubernetes.io/projected/30cd348d-5e87-4114-a53c-ffb9095bd1d8-kube-api-access-gfv2j\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.619663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.619645 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.620161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.620137 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.621974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.621954 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.644343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.644316 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfv2j\" (UniqueName: \"kubernetes.io/projected/30cd348d-5e87-4114-a53c-ffb9095bd1d8-kube-api-access-gfv2j\") pod \"maas-api-revoke-sub-c1f0e314-f7m96\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:56.715694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:56.715661 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:57.033855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.033822 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96"] Nov 02 14:11:57.035144 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:57.035122 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cd348d_5e87_4114_a53c_ffb9095bd1d8.slice/crio-f419fdcdeb3e19d3173197e295a9d032478f222d4cc5bfb6a4a79340631cc3d4 WatchSource:0}: Error finding container f419fdcdeb3e19d3173197e295a9d032478f222d4cc5bfb6a4a79340631cc3d4: Status 404 returned error can't find the container with id f419fdcdeb3e19d3173197e295a9d032478f222d4cc5bfb6a4a79340631cc3d4 Nov 02 14:11:57.198804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.198736 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q"] Nov 02 14:11:57.203633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.203610 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.218592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.218572 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q"] Nov 02 14:11:57.227209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.227180 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.227312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.227232 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fr6m\" (UniqueName: \"kubernetes.io/projected/61f58be5-07f2-4e8d-b619-98cb13afe2b5-kube-api-access-2fr6m\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.227382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.227318 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.245461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.245438 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:11:57.245569 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.245488 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 14:11:57.327875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.327759 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.327875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.327853 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fr6m\" (UniqueName: \"kubernetes.io/projected/61f58be5-07f2-4e8d-b619-98cb13afe2b5-kube-api-access-2fr6m\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.328375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.327928 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.328552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.328534 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.330089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.330064 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.347335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.347307 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" event={"ID":"30cd348d-5e87-4114-a53c-ffb9095bd1d8","Type":"ContainerStarted","Data":"f419fdcdeb3e19d3173197e295a9d032478f222d4cc5bfb6a4a79340631cc3d4"} Nov 02 14:11:57.352982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.352961 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fr6m\" (UniqueName: \"kubernetes.io/projected/61f58be5-07f2-4e8d-b619-98cb13afe2b5-kube-api-access-2fr6m\") pod \"maas-api-revoke-sub-c9b654dd-pjg2q\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.532754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.532722 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:11:57.864563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.864531 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q"] Nov 02 14:11:57.866381 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:11:57.866355 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f58be5_07f2_4e8d_b619_98cb13afe2b5.slice/crio-b02383067d5fdef5d531d42f3a4179e8fe6e1c7591963b67d66f27d15644fc90 WatchSource:0}: Error finding container b02383067d5fdef5d531d42f3a4179e8fe6e1c7591963b67d66f27d15644fc90: Status 404 returned error can't find the container with id b02383067d5fdef5d531d42f3a4179e8fe6e1c7591963b67d66f27d15644fc90 Nov 02 14:11:57.906143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:57.906111 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:11:58.246059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:58.246033 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:11:58.246141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:58.246085 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 14:11:58.355545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:58.355509 4431 generic.go:356] "Generic (PLEG): container finished" podID="30cd348d-5e87-4114-a53c-ffb9095bd1d8" containerID="726cfc75679367eb55e413331caff34e4ecc1a7327b0dfcb2b785ee1d99254eb" exitCode=0 Nov 02 14:11:58.355974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:58.355605 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" event={"ID":"30cd348d-5e87-4114-a53c-ffb9095bd1d8","Type":"ContainerDied","Data":"726cfc75679367eb55e413331caff34e4ecc1a7327b0dfcb2b785ee1d99254eb"} Nov 02 14:11:58.357478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:58.357456 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" event={"ID":"61f58be5-07f2-4e8d-b619-98cb13afe2b5","Type":"ContainerStarted","Data":"c25c5e858606a29ffef332d782b0fce4e4665029270b8134b4e385254c22fac3"} Nov 02 14:11:58.357615 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:58.357499 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" event={"ID":"61f58be5-07f2-4e8d-b619-98cb13afe2b5","Type":"ContainerStarted","Data":"b02383067d5fdef5d531d42f3a4179e8fe6e1c7591963b67d66f27d15644fc90"} Nov 02 14:11:58.516843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:58.515928 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" podStartSLOduration=1.5159091770000002 podStartE2EDuration="1.515909177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:58.513113324 +0000 UTC m=+883.550218253" watchObservedRunningTime="2026-11-02 14:11:58.515909177 +0000 UTC m=+883.553014112" Nov 02 14:11:59.257284 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.257254 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 14:11:59.257474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.257331 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 14:11:59.368557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.368456 4431 generic.go:356] "Generic (PLEG): container finished" podID="61f58be5-07f2-4e8d-b619-98cb13afe2b5" containerID="c25c5e858606a29ffef332d782b0fce4e4665029270b8134b4e385254c22fac3" exitCode=0 Nov 02 14:11:59.368557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.368539 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" event={"ID":"61f58be5-07f2-4e8d-b619-98cb13afe2b5","Type":"ContainerDied","Data":"c25c5e858606a29ffef332d782b0fce4e4665029270b8134b4e385254c22fac3"} Nov 02 14:11:59.727516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.727495 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:11:59.767417 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.767395 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-openshift-ca\") pod \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " Nov 02 14:11:59.767569 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.767481 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-xks-ca\") pod \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " Nov 02 14:11:59.767569 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.767542 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cd348d-5e87-4114-a53c-ffb9095bd1d8-kube-api-access-gfv2j\" (UniqueName: \"kubernetes.io/projected/30cd348d-5e87-4114-a53c-ffb9095bd1d8-kube-api-access-gfv2j\") pod \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\" (UID: \"30cd348d-5e87-4114-a53c-ffb9095bd1d8\") " Nov 02 14:11:59.767770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.767740 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-openshift-ca" pod "30cd348d-5e87-4114-a53c-ffb9095bd1d8" (UID: "30cd348d-5e87-4114-a53c-ffb9095bd1d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:11:59.769605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.769579 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-xks-ca" pod "30cd348d-5e87-4114-a53c-ffb9095bd1d8" (UID: "30cd348d-5e87-4114-a53c-ffb9095bd1d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:11:59.769705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.769678 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cd348d-5e87-4114-a53c-ffb9095bd1d8-kube-api-access-gfv2j" pod "30cd348d-5e87-4114-a53c-ffb9095bd1d8" (UID: "30cd348d-5e87-4114-a53c-ffb9095bd1d8"). InnerVolumeSpecName "kube-api-access-gfv2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:11:59.869351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.869297 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:59.869351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.869347 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30cd348d-5e87-4114-a53c-ffb9095bd1d8-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:11:59.869351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:11:59.869357 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfv2j\" (UniqueName: \"kubernetes.io/projected/30cd348d-5e87-4114-a53c-ffb9095bd1d8-kube-api-access-gfv2j\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.316478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.316442 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:12:00.316959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.316935 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:12:00.317639 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.317616 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:12:00.376410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.376384 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" Nov 02 14:12:00.376811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.376419 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96" event={"ID":"30cd348d-5e87-4114-a53c-ffb9095bd1d8","Type":"ContainerDied","Data":"f419fdcdeb3e19d3173197e295a9d032478f222d4cc5bfb6a4a79340631cc3d4"} Nov 02 14:12:00.376811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.376473 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f419fdcdeb3e19d3173197e295a9d032478f222d4cc5bfb6a4a79340631cc3d4" Nov 02 14:12:00.496792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.496746 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7"] Nov 02 14:12:00.497489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.497465 4431 cpu_manager.go:414] "containerMap: removing container" podUID="30cd348d-5e87-4114-a53c-ffb9095bd1d8" containerName="revoke-keys" Nov 02 14:12:00.497489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.497486 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cd348d-5e87-4114-a53c-ffb9095bd1d8" containerName="revoke-keys" Nov 02 14:12:00.497673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.497596 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cd348d-5e87-4114-a53c-ffb9095bd1d8" containerName="revoke-keys" Nov 02 14:12:00.501494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.501469 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.534949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.534926 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7"] Nov 02 14:12:00.558682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.558661 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:12:00.584470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.583187 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f58be5-07f2-4e8d-b619-98cb13afe2b5-kube-api-access-2fr6m\" (UniqueName: \"kubernetes.io/projected/61f58be5-07f2-4e8d-b619-98cb13afe2b5-kube-api-access-2fr6m\") pod \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " Nov 02 14:12:00.584470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.583256 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-xks-ca\") pod \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " Nov 02 14:12:00.584470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.583338 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-openshift-ca\") pod \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\" (UID: \"61f58be5-07f2-4e8d-b619-98cb13afe2b5\") " Nov 02 14:12:00.584470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.583765 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4s7\" (UniqueName: \"kubernetes.io/projected/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-kube-api-access-7p4s7\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.584470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.583883 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.584470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.584004 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.586332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.586295 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-openshift-ca" pod "61f58be5-07f2-4e8d-b619-98cb13afe2b5" (UID: "61f58be5-07f2-4e8d-b619-98cb13afe2b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:00.587443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.587386 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f58be5-07f2-4e8d-b619-98cb13afe2b5-kube-api-access-2fr6m" pod "61f58be5-07f2-4e8d-b619-98cb13afe2b5" (UID: "61f58be5-07f2-4e8d-b619-98cb13afe2b5"). InnerVolumeSpecName "kube-api-access-2fr6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:00.587565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.587535 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-xks-ca" pod "61f58be5-07f2-4e8d-b619-98cb13afe2b5" (UID: "61f58be5-07f2-4e8d-b619-98cb13afe2b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:00.685371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.685327 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4s7\" (UniqueName: \"kubernetes.io/projected/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-kube-api-access-7p4s7\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.685553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.685446 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.685621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.685566 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.685673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.685663 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.685730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.685682 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f58be5-07f2-4e8d-b619-98cb13afe2b5-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.685730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.685699 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fr6m\" (UniqueName: \"kubernetes.io/projected/61f58be5-07f2-4e8d-b619-98cb13afe2b5-kube-api-access-2fr6m\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:00.686255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.686229 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.687771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.687741 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.713905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.713881 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4s7\" (UniqueName: \"kubernetes.io/projected/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-kube-api-access-7p4s7\") pod \"maas-api-revoke-sub-080004d2-8h5h7\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:00.854088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:00.854020 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:01.170625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:01.170603 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7"] Nov 02 14:12:01.172376 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:01.172352 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaaa5a9_9ae2_4b85_83f4_46304acb5005.slice/crio-f31556facc0d0cd8343c750904ad524fc57c7242f8bb8591f5daa4e2f4e126e6 WatchSource:0}: Error finding container f31556facc0d0cd8343c750904ad524fc57c7242f8bb8591f5daa4e2f4e126e6: Status 404 returned error can't find the container with id f31556facc0d0cd8343c750904ad524fc57c7242f8bb8591f5daa4e2f4e126e6 Nov 02 14:12:01.384576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:01.384547 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" event={"ID":"4aaaa5a9-9ae2-4b85-83f4-46304acb5005","Type":"ContainerStarted","Data":"f31556facc0d0cd8343c750904ad524fc57c7242f8bb8591f5daa4e2f4e126e6"} Nov 02 14:12:01.385976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:01.385947 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" event={"ID":"61f58be5-07f2-4e8d-b619-98cb13afe2b5","Type":"ContainerDied","Data":"b02383067d5fdef5d531d42f3a4179e8fe6e1c7591963b67d66f27d15644fc90"} Nov 02 14:12:01.385976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:01.385966 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q" Nov 02 14:12:01.386137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:01.385997 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02383067d5fdef5d531d42f3a4179e8fe6e1c7591963b67d66f27d15644fc90" Nov 02 14:12:02.392861 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:02.392826 4431 generic.go:356] "Generic (PLEG): container finished" podID="4aaaa5a9-9ae2-4b85-83f4-46304acb5005" containerID="641bf69f215ae0ee071d1c3f0d50bebfaafbaaadea1b88e62103d2293c518435" exitCode=0 Nov 02 14:12:02.393213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:02.392883 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" event={"ID":"4aaaa5a9-9ae2-4b85-83f4-46304acb5005","Type":"ContainerDied","Data":"641bf69f215ae0ee071d1c3f0d50bebfaafbaaadea1b88e62103d2293c518435"} Nov 02 14:12:03.542184 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.542157 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:03.614579 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.614524 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-kube-api-access-7p4s7\" (UniqueName: \"kubernetes.io/projected/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-kube-api-access-7p4s7\") pod \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " Nov 02 14:12:03.614801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.614624 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-xks-ca\") pod \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " Nov 02 14:12:03.614801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.614716 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-openshift-ca\") pod \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\" (UID: \"4aaaa5a9-9ae2-4b85-83f4-46304acb5005\") " Nov 02 14:12:03.615702 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.615678 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-openshift-ca" pod "4aaaa5a9-9ae2-4b85-83f4-46304acb5005" (UID: "4aaaa5a9-9ae2-4b85-83f4-46304acb5005"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:03.617493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.617466 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-xks-ca" pod "4aaaa5a9-9ae2-4b85-83f4-46304acb5005" (UID: "4aaaa5a9-9ae2-4b85-83f4-46304acb5005"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:03.617669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.617649 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-kube-api-access-7p4s7" pod "4aaaa5a9-9ae2-4b85-83f4-46304acb5005" (UID: "4aaaa5a9-9ae2-4b85-83f4-46304acb5005"). InnerVolumeSpecName "kube-api-access-7p4s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:03.715972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.715913 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4s7\" (UniqueName: \"kubernetes.io/projected/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-kube-api-access-7p4s7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:03.715972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.715937 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:03.715972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:03.715946 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aaaa5a9-9ae2-4b85-83f4-46304acb5005-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:04.405104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:04.405074 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" Nov 02 14:12:04.405104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:04.405090 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7" event={"ID":"4aaaa5a9-9ae2-4b85-83f4-46304acb5005","Type":"ContainerDied","Data":"f31556facc0d0cd8343c750904ad524fc57c7242f8bb8591f5daa4e2f4e126e6"} Nov 02 14:12:04.405310 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:04.405134 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31556facc0d0cd8343c750904ad524fc57c7242f8bb8591f5daa4e2f4e126e6" Nov 02 14:12:11.076558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.076522 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm"] Nov 02 14:12:11.077091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.077074 4431 cpu_manager.go:414] "containerMap: removing container" podUID="61f58be5-07f2-4e8d-b619-98cb13afe2b5" containerName="revoke-keys" Nov 02 14:12:11.077143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.077093 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f58be5-07f2-4e8d-b619-98cb13afe2b5" containerName="revoke-keys" Nov 02 14:12:11.077143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.077106 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4aaaa5a9-9ae2-4b85-83f4-46304acb5005" containerName="revoke-keys" Nov 02 14:12:11.077143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.077112 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aaaa5a9-9ae2-4b85-83f4-46304acb5005" containerName="revoke-keys" Nov 02 14:12:11.077258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.077181 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f58be5-07f2-4e8d-b619-98cb13afe2b5" containerName="revoke-keys" Nov 02 14:12:11.077258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.077188 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aaaa5a9-9ae2-4b85-83f4-46304acb5005" containerName="revoke-keys" Nov 02 14:12:11.080993 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.080970 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.094840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.094761 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm"] Nov 02 14:12:11.107552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.107523 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:12:11.107675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.107575 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:12:11.175370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.175335 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347c113b-213b-40f6-b92a-91005869198b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.176694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.176667 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6982\" (UniqueName: \"kubernetes.io/projected/347c113b-213b-40f6-b92a-91005869198b-kube-api-access-k6982\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.176875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.176843 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347c113b-213b-40f6-b92a-91005869198b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.277697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.277651 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6982\" (UniqueName: \"kubernetes.io/projected/347c113b-213b-40f6-b92a-91005869198b-kube-api-access-k6982\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.277886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.277737 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347c113b-213b-40f6-b92a-91005869198b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.277886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.277810 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347c113b-213b-40f6-b92a-91005869198b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.278616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.278587 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347c113b-213b-40f6-b92a-91005869198b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.280475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.280453 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347c113b-213b-40f6-b92a-91005869198b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.308392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.308362 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6982\" (UniqueName: \"kubernetes.io/projected/347c113b-213b-40f6-b92a-91005869198b-kube-api-access-k6982\") pod \"maas-api-revoke-sub-1b20bd8f-99srm\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.395509 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.395487 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:11.710761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:11.710739 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm"] Nov 02 14:12:11.712698 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:11.712665 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347c113b_213b_40f6_b92a_91005869198b.slice/crio-e16f83552b672dee9f60a90411c38321829531c59ec4fcb56eac567e1a5486f3 WatchSource:0}: Error finding container e16f83552b672dee9f60a90411c38321829531c59ec4fcb56eac567e1a5486f3: Status 404 returned error can't find the container with id e16f83552b672dee9f60a90411c38321829531c59ec4fcb56eac567e1a5486f3 Nov 02 14:12:12.465511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:12.465477 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" event={"ID":"347c113b-213b-40f6-b92a-91005869198b","Type":"ContainerStarted","Data":"55186ef8e940674e52668e5f016eec09d005c5d418bc610ca53ae9f7c24945db"} Nov 02 14:12:12.465887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:12.465533 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" event={"ID":"347c113b-213b-40f6-b92a-91005869198b","Type":"ContainerStarted","Data":"e16f83552b672dee9f60a90411c38321829531c59ec4fcb56eac567e1a5486f3"} Nov 02 14:12:12.513187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:12.513139 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" podStartSLOduration=1.513126127 podStartE2EDuration="1.513126127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:12.511855029 +0000 UTC m=+897.548959961" watchObservedRunningTime="2026-11-02 14:12:12.513126127 +0000 UTC m=+897.550231058" Nov 02 14:12:13.472423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:13.472391 4431 generic.go:356] "Generic (PLEG): container finished" podID="347c113b-213b-40f6-b92a-91005869198b" containerID="55186ef8e940674e52668e5f016eec09d005c5d418bc610ca53ae9f7c24945db" exitCode=0 Nov 02 14:12:13.472851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:13.472478 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" event={"ID":"347c113b-213b-40f6-b92a-91005869198b","Type":"ContainerDied","Data":"55186ef8e940674e52668e5f016eec09d005c5d418bc610ca53ae9f7c24945db"} Nov 02 14:12:14.623748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.623722 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:14.715803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.715752 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347c113b-213b-40f6-b92a-91005869198b-kube-api-access-k6982\" (UniqueName: \"kubernetes.io/projected/347c113b-213b-40f6-b92a-91005869198b-kube-api-access-k6982\") pod \"347c113b-213b-40f6-b92a-91005869198b\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " Nov 02 14:12:14.715940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.715863 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347c113b-213b-40f6-b92a-91005869198b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347c113b-213b-40f6-b92a-91005869198b-maas-api-openshift-ca\") pod \"347c113b-213b-40f6-b92a-91005869198b\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " Nov 02 14:12:14.715982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.715944 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347c113b-213b-40f6-b92a-91005869198b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347c113b-213b-40f6-b92a-91005869198b-maas-api-xks-ca\") pod \"347c113b-213b-40f6-b92a-91005869198b\" (UID: \"347c113b-213b-40f6-b92a-91005869198b\") " Nov 02 14:12:14.716261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.716234 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347c113b-213b-40f6-b92a-91005869198b-maas-api-openshift-ca" pod "347c113b-213b-40f6-b92a-91005869198b" (UID: "347c113b-213b-40f6-b92a-91005869198b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:14.717631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.717611 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347c113b-213b-40f6-b92a-91005869198b-kube-api-access-k6982" pod "347c113b-213b-40f6-b92a-91005869198b" (UID: "347c113b-213b-40f6-b92a-91005869198b"). InnerVolumeSpecName "kube-api-access-k6982". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:14.717747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.717732 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347c113b-213b-40f6-b92a-91005869198b-maas-api-xks-ca" pod "347c113b-213b-40f6-b92a-91005869198b" (UID: "347c113b-213b-40f6-b92a-91005869198b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:14.817475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.817416 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347c113b-213b-40f6-b92a-91005869198b-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:14.817475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.817438 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6982\" (UniqueName: \"kubernetes.io/projected/347c113b-213b-40f6-b92a-91005869198b-kube-api-access-k6982\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:14.817475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:14.817448 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347c113b-213b-40f6-b92a-91005869198b-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:15.485100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:15.485074 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" Nov 02 14:12:15.485257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:15.485077 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm" event={"ID":"347c113b-213b-40f6-b92a-91005869198b","Type":"ContainerDied","Data":"e16f83552b672dee9f60a90411c38321829531c59ec4fcb56eac567e1a5486f3"} Nov 02 14:12:15.485257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:15.485199 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e16f83552b672dee9f60a90411c38321829531c59ec4fcb56eac567e1a5486f3" Nov 02 14:12:17.737012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:17.736969 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:12:21.283697 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:12:21.283655 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:12:21.433959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.433926 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg"] Nov 02 14:12:21.434406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.434394 4431 cpu_manager.go:414] "containerMap: removing container" podUID="347c113b-213b-40f6-b92a-91005869198b" containerName="revoke-keys" Nov 02 14:12:21.434449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.434407 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347c113b-213b-40f6-b92a-91005869198b" containerName="revoke-keys" Nov 02 14:12:21.434494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.434486 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="347c113b-213b-40f6-b92a-91005869198b" containerName="revoke-keys" Nov 02 14:12:21.438963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.438948 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.458452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.458423 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg"] Nov 02 14:12:21.467582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.467558 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:12:21.467697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.467603 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:12:21.520473 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.520449 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbltx\" (UniqueName: \"kubernetes.io/projected/4b856e82-d30a-4729-a3ab-a45059046610-kube-api-access-rbltx\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.520595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.520480 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.520644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.520611 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.622130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.622068 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbltx\" (UniqueName: \"kubernetes.io/projected/4b856e82-d30a-4729-a3ab-a45059046610-kube-api-access-rbltx\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.622130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.622105 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.622276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.622178 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.623016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.622977 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.624810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.624758 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.647434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.647410 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbltx\" (UniqueName: \"kubernetes.io/projected/4b856e82-d30a-4729-a3ab-a45059046610-kube-api-access-rbltx\") pod \"maas-api-revoke-sub-4a0e4c6c-cmrtg\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:21.751882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:21.751860 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:22.051698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.051669 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg"] Nov 02 14:12:22.052012 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:22.051990 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b856e82_d30a_4729_a3ab_a45059046610.slice/crio-36eb6d676ee53f95a500cca726ab89f72ea045e6c2747f3c9f65e0c8d4245b75 WatchSource:0}: Error finding container 36eb6d676ee53f95a500cca726ab89f72ea045e6c2747f3c9f65e0c8d4245b75: Status 404 returned error can't find the container with id 36eb6d676ee53f95a500cca726ab89f72ea045e6c2747f3c9f65e0c8d4245b75 Nov 02 14:12:22.539624 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.539588 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" event={"ID":"4b856e82-d30a-4729-a3ab-a45059046610","Type":"ContainerStarted","Data":"8dfa06455dee55835bf32aad8a2c5151ac9d3c8476219c79d67331ac2a6fbdfd"} Nov 02 14:12:22.540059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.539668 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" event={"ID":"4b856e82-d30a-4729-a3ab-a45059046610","Type":"ContainerStarted","Data":"36eb6d676ee53f95a500cca726ab89f72ea045e6c2747f3c9f65e0c8d4245b75"} Nov 02 14:12:22.585056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.585011 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" podStartSLOduration=1.584998701 podStartE2EDuration="1.584998701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:22.58258197 +0000 UTC m=+907.619686905" watchObservedRunningTime="2026-11-02 14:12:22.584998701 +0000 UTC m=+907.622103634" Nov 02 14:12:22.885097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.885069 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq"] Nov 02 14:12:22.888882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.888867 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.923970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.923945 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq"] Nov 02 14:12:22.935465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.935441 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.935654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.935627 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afbe79c-479d-46f5-9d36-d1443461695f-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.935762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.935676 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.935848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.935813 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.935897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.935847 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.935945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.935930 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.935991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.935960 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7mt\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-kube-api-access-7w7mt\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.936027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.936013 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:22.936061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:22.936029 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afbe79c-479d-46f5-9d36-d1443461695f-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037253 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037296 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037345 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037371 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7mt\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-kube-api-access-7w7mt\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037498 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037540 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afbe79c-479d-46f5-9d36-d1443461695f-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037610 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037768 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afbe79c-479d-46f5-9d36-d1443461695f-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037831 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-certs\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037859 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.037903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.037867 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-data\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.038063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.038004 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.038238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.038214 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-credential-socket\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.038310 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.038275 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afbe79c-479d-46f5-9d36-d1443461695f-istiod-ca-cert\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.039445 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.039424 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-envoy\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.040145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.040124 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afbe79c-479d-46f5-9d36-d1443461695f-istio-podinfo\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.062352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.062301 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-istio-token\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.062814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.062793 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7mt\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-kube-api-access-7w7mt\") pod \"e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.164341 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.164315 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5"] Nov 02 14:12:23.169307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.169286 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.195044 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.195018 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5"] Nov 02 14:12:23.203341 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.203320 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:23.239624 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.239599 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.239787 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.239634 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.239787 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.239742 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.239929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.239874 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.239983 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.239929 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qvn\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-kube-api-access-78qvn\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.240037 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.239982 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccf69b9-f541-4987-b45a-649d076d545a-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.240095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.240067 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccf69b9-f541-4987-b45a-649d076d545a-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.240153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.240121 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.240207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.240171 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341051 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341087 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341129 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341153 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341178 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qvn\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-kube-api-access-78qvn\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341210 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccf69b9-f541-4987-b45a-649d076d545a-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341315 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccf69b9-f541-4987-b45a-649d076d545a-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341379 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341435 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341595 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341644 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341701 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.341907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.341840 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.342123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.342104 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccf69b9-f541-4987-b45a-649d076d545a-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.343551 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.343533 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.343680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.343664 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccf69b9-f541-4987-b45a-649d076d545a-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.366017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.365989 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.366123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.366108 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qvn\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-kube-api-access-78qvn\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.484074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.484048 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:23.528332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.528297 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq"] Nov 02 14:12:23.529933 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:23.529896 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afbe79c_479d_46f5_9d36_d1443461695f.slice/crio-cf81a6518e5ef8cc33869e15bbd0edabc69e52486df28a5024a6af33d0eb4eac WatchSource:0}: Error finding container cf81a6518e5ef8cc33869e15bbd0edabc69e52486df28a5024a6af33d0eb4eac: Status 404 returned error can't find the container with id cf81a6518e5ef8cc33869e15bbd0edabc69e52486df28a5024a6af33d0eb4eac Nov 02 14:12:23.532297 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.532262 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:23.532398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.532327 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:23.532398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.532368 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:23.552032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.550415 4431 generic.go:356] "Generic (PLEG): container finished" podID="4b856e82-d30a-4729-a3ab-a45059046610" containerID="8dfa06455dee55835bf32aad8a2c5151ac9d3c8476219c79d67331ac2a6fbdfd" exitCode=0 Nov 02 14:12:23.552032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.550882 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" event={"ID":"4b856e82-d30a-4729-a3ab-a45059046610","Type":"ContainerDied","Data":"8dfa06455dee55835bf32aad8a2c5151ac9d3c8476219c79d67331ac2a6fbdfd"} Nov 02 14:12:23.552682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.552644 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" event={"ID":"3afbe79c-479d-46f5-9d36-d1443461695f","Type":"ContainerStarted","Data":"cf81a6518e5ef8cc33869e15bbd0edabc69e52486df28a5024a6af33d0eb4eac"} Nov 02 14:12:23.791401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.791375 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5"] Nov 02 14:12:23.792631 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:23.792604 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccf69b9_f541_4987_b45a_649d076d545a.slice/crio-e6a882b978621b6a02340e0e86a45f40c16f7f5241ed67515c899d6eb99dfb73 WatchSource:0}: Error finding container e6a882b978621b6a02340e0e86a45f40c16f7f5241ed67515c899d6eb99dfb73: Status 404 returned error can't find the container with id e6a882b978621b6a02340e0e86a45f40c16f7f5241ed67515c899d6eb99dfb73 Nov 02 14:12:23.795700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.795655 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:23.795848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.795757 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:23.795910 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:23.795842 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:24.562454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.562422 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" event={"ID":"3afbe79c-479d-46f5-9d36-d1443461695f","Type":"ContainerStarted","Data":"41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318"} Nov 02 14:12:24.564374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.564349 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" event={"ID":"cccf69b9-f541-4987-b45a-649d076d545a","Type":"ContainerStarted","Data":"c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a"} Nov 02 14:12:24.564507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.564388 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" event={"ID":"cccf69b9-f541-4987-b45a-649d076d545a","Type":"ContainerStarted","Data":"e6a882b978621b6a02340e0e86a45f40c16f7f5241ed67515c899d6eb99dfb73"} Nov 02 14:12:24.630524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.629454 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podStartSLOduration=2.629434517 podStartE2EDuration="2.629434517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:24.624444706 +0000 UTC m=+909.661549638" watchObservedRunningTime="2026-11-02 14:12:24.629434517 +0000 UTC m=+909.666539450" Nov 02 14:12:24.921826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.921391 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:24.959856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.959825 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-openshift-ca\") pod \"4b856e82-d30a-4729-a3ab-a45059046610\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " Nov 02 14:12:24.960017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.959935 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-xks-ca\") pod \"4b856e82-d30a-4729-a3ab-a45059046610\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " Nov 02 14:12:24.960069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.960050 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b856e82-d30a-4729-a3ab-a45059046610-kube-api-access-rbltx\" (UniqueName: \"kubernetes.io/projected/4b856e82-d30a-4729-a3ab-a45059046610-kube-api-access-rbltx\") pod \"4b856e82-d30a-4729-a3ab-a45059046610\" (UID: \"4b856e82-d30a-4729-a3ab-a45059046610\") " Nov 02 14:12:24.960710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.960687 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-openshift-ca" pod "4b856e82-d30a-4729-a3ab-a45059046610" (UID: "4b856e82-d30a-4729-a3ab-a45059046610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:24.962120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.962099 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b856e82-d30a-4729-a3ab-a45059046610-kube-api-access-rbltx" pod "4b856e82-d30a-4729-a3ab-a45059046610" (UID: "4b856e82-d30a-4729-a3ab-a45059046610"). InnerVolumeSpecName "kube-api-access-rbltx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:24.962402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:24.962388 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-xks-ca" pod "4b856e82-d30a-4729-a3ab-a45059046610" (UID: "4b856e82-d30a-4729-a3ab-a45059046610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:25.044799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.044172 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" podStartSLOduration=2.044149782 podStartE2EDuration="2.044149782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:24.698732852 +0000 UTC m=+909.735837796" watchObservedRunningTime="2026-11-02 14:12:25.044149782 +0000 UTC m=+910.081254716" Nov 02 14:12:25.164968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.164463 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:25.164968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.164496 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b856e82-d30a-4729-a3ab-a45059046610-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:25.164968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.164519 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbltx\" (UniqueName: \"kubernetes.io/projected/4b856e82-d30a-4729-a3ab-a45059046610-kube-api-access-rbltx\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:25.223076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.215361 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:25.225023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.224996 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:25.225213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.225188 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 14:12:25.491577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.490829 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:25.491873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.491850 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:25.492046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.492021 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 14:12:25.604802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.600923 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" Nov 02 14:12:25.604802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.601424 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg" event={"ID":"4b856e82-d30a-4729-a3ab-a45059046610","Type":"ContainerDied","Data":"36eb6d676ee53f95a500cca726ab89f72ea045e6c2747f3c9f65e0c8d4245b75"} Nov 02 14:12:25.604802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:25.601464 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36eb6d676ee53f95a500cca726ab89f72ea045e6c2747f3c9f65e0c8d4245b75" Nov 02 14:12:26.207806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:26.207034 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:26.207806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:26.207090 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 14:12:26.490856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:26.490376 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:26.490856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:26.490527 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 14:12:27.209801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:27.204182 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:27.209801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:27.204243 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 14:12:27.489858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:27.486203 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:27.489858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:27.486263 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 14:12:28.206802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:28.206155 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:28.206802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:28.206217 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 14:12:28.491860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:28.491160 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:28.491860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:28.491223 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 14:12:28.997801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:28.997135 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf"] Nov 02 14:12:28.998013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:28.997989 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4b856e82-d30a-4729-a3ab-a45059046610" containerName="revoke-keys" Nov 02 14:12:28.998086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:28.998012 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b856e82-d30a-4729-a3ab-a45059046610" containerName="revoke-keys" Nov 02 14:12:28.998804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:28.998142 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b856e82-d30a-4729-a3ab-a45059046610" containerName="revoke-keys" Nov 02 14:12:29.028797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.028140 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf"] Nov 02 14:12:29.028797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.028285 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.044508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.044478 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:12:29.054133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.054112 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zm42\" (UniqueName: \"kubernetes.io/projected/938f348b-7d83-4670-b488-eb500446e69d-kube-api-access-2zm42\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.054248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.054228 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f348b-7d83-4670-b488-eb500446e69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.054304 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.054294 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f348b-7d83-4670-b488-eb500446e69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.057589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.057570 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:12:29.155248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.155213 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f348b-7d83-4670-b488-eb500446e69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.155440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.155388 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zm42\" (UniqueName: \"kubernetes.io/projected/938f348b-7d83-4670-b488-eb500446e69d-kube-api-access-2zm42\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.155509 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.155463 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f348b-7d83-4670-b488-eb500446e69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.155981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.155957 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f348b-7d83-4670-b488-eb500446e69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.157666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.157648 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f348b-7d83-4670-b488-eb500446e69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.191454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.191427 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zm42\" (UniqueName: \"kubernetes.io/projected/938f348b-7d83-4670-b488-eb500446e69d-kube-api-access-2zm42\") pod \"maas-api-revoke-sub-47b68a13-8sttf\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.204147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.204119 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:29.204366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.204338 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 14:12:29.343617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.343543 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:29.497834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.492984 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 02 14:12:29.497834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.493054 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 02 14:12:29.708153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:29.708108 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf"] Nov 02 14:12:29.710904 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:29.710878 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938f348b_7d83_4670_b488_eb500446e69d.slice/crio-961895e4cab38bbd626c276105d8944ef13b3f95f954d5dfdea34b56db235bab WatchSource:0}: Error finding container 961895e4cab38bbd626c276105d8944ef13b3f95f954d5dfdea34b56db235bab: Status 404 returned error can't find the container with id 961895e4cab38bbd626c276105d8944ef13b3f95f954d5dfdea34b56db235bab Nov 02 14:12:30.204023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:30.203953 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 14:12:30.204197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:30.204021 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 14:12:30.599469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:30.599441 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:30.599940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:30.599922 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:30.600890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:30.600869 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:12:30.663703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:30.663660 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" event={"ID":"938f348b-7d83-4670-b488-eb500446e69d","Type":"ContainerStarted","Data":"45379739ae82042ba175f332a7a2726e720ec8208a622f9e8653d15e491b4a16"} Nov 02 14:12:30.663900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:30.663731 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" event={"ID":"938f348b-7d83-4670-b488-eb500446e69d","Type":"ContainerStarted","Data":"961895e4cab38bbd626c276105d8944ef13b3f95f954d5dfdea34b56db235bab"} Nov 02 14:12:30.715805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:30.714761 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" podStartSLOduration=2.714743861 podStartE2EDuration="2.714743861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:30.711941005 +0000 UTC m=+915.749045940" watchObservedRunningTime="2026-11-02 14:12:30.714743861 +0000 UTC m=+915.751848794" Nov 02 14:12:31.208099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:31.208071 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:31.208462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:31.208440 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:31.209029 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:31.209014 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:12:31.671829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:31.671771 4431 generic.go:356] "Generic (PLEG): container finished" podID="938f348b-7d83-4670-b488-eb500446e69d" containerID="45379739ae82042ba175f332a7a2726e720ec8208a622f9e8653d15e491b4a16" exitCode=0 Nov 02 14:12:31.671829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:31.671812 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" event={"ID":"938f348b-7d83-4670-b488-eb500446e69d","Type":"ContainerDied","Data":"45379739ae82042ba175f332a7a2726e720ec8208a622f9e8653d15e491b4a16"} Nov 02 14:12:33.905299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:33.905268 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:33.977496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:33.977412 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938f348b-7d83-4670-b488-eb500446e69d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f348b-7d83-4670-b488-eb500446e69d-maas-api-xks-ca\") pod \"938f348b-7d83-4670-b488-eb500446e69d\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " Nov 02 14:12:34.006309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.006275 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938f348b-7d83-4670-b488-eb500446e69d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f348b-7d83-4670-b488-eb500446e69d-maas-api-openshift-ca\") pod \"938f348b-7d83-4670-b488-eb500446e69d\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " Nov 02 14:12:34.007189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.007160 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938f348b-7d83-4670-b488-eb500446e69d-kube-api-access-2zm42\" (UniqueName: \"kubernetes.io/projected/938f348b-7d83-4670-b488-eb500446e69d-kube-api-access-2zm42\") pod \"938f348b-7d83-4670-b488-eb500446e69d\" (UID: \"938f348b-7d83-4670-b488-eb500446e69d\") " Nov 02 14:12:34.013884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.007041 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938f348b-7d83-4670-b488-eb500446e69d-maas-api-openshift-ca" pod "938f348b-7d83-4670-b488-eb500446e69d" (UID: "938f348b-7d83-4670-b488-eb500446e69d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:34.014653 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.014531 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938f348b-7d83-4670-b488-eb500446e69d-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:34.066807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.035346 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938f348b-7d83-4670-b488-eb500446e69d-kube-api-access-2zm42" pod "938f348b-7d83-4670-b488-eb500446e69d" (UID: "938f348b-7d83-4670-b488-eb500446e69d"). InnerVolumeSpecName "kube-api-access-2zm42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:34.066807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.064696 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938f348b-7d83-4670-b488-eb500446e69d-maas-api-xks-ca" pod "938f348b-7d83-4670-b488-eb500446e69d" (UID: "938f348b-7d83-4670-b488-eb500446e69d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:34.120054 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.120023 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4"] Nov 02 14:12:34.121013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.120988 4431 cpu_manager.go:414] "containerMap: removing container" podUID="938f348b-7d83-4670-b488-eb500446e69d" containerName="revoke-keys" Nov 02 14:12:34.121161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.121145 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938f348b-7d83-4670-b488-eb500446e69d" containerName="revoke-keys" Nov 02 14:12:34.121385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.121370 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="938f348b-7d83-4670-b488-eb500446e69d" containerName="revoke-keys" Nov 02 14:12:34.401799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.369682 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938f348b-7d83-4670-b488-eb500446e69d-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:34.402028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.402006 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zm42\" (UniqueName: \"kubernetes.io/projected/938f348b-7d83-4670-b488-eb500446e69d-kube-api-access-2zm42\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:34.438807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.420796 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4"] Nov 02 14:12:34.439057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.428944 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.541802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.530521 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.541802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.530583 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwcw\" (UniqueName: \"kubernetes.io/projected/641b4904-b6de-421c-b5a3-c6ab2a5fc006-kube-api-access-ngwcw\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.541802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.530629 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.632600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.632570 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.633767 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.633744 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwcw\" (UniqueName: \"kubernetes.io/projected/641b4904-b6de-421c-b5a3-c6ab2a5fc006-kube-api-access-ngwcw\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.633962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.633947 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.634970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.634947 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.640150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.640105 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.665045 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.663355 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwcw\" (UniqueName: \"kubernetes.io/projected/641b4904-b6de-421c-b5a3-c6ab2a5fc006-kube-api-access-ngwcw\") pod \"maas-api-revoke-sub-442194a2-tstn4\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:34.898913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.898883 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" event={"ID":"938f348b-7d83-4670-b488-eb500446e69d","Type":"ContainerDied","Data":"961895e4cab38bbd626c276105d8944ef13b3f95f954d5dfdea34b56db235bab"} Nov 02 14:12:34.899062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.898928 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="961895e4cab38bbd626c276105d8944ef13b3f95f954d5dfdea34b56db235bab" Nov 02 14:12:34.899062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.898926 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf" Nov 02 14:12:34.909894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:34.909873 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:35.222725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:35.222694 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4"] Nov 02 14:12:35.223114 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:35.223087 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641b4904_b6de_421c_b5a3_c6ab2a5fc006.slice/crio-a7ccd7e2ab1ef8fad07f2ea1273a5eb0353c09c15f6883492f911a9a1b9ac3dd WatchSource:0}: Error finding container a7ccd7e2ab1ef8fad07f2ea1273a5eb0353c09c15f6883492f911a9a1b9ac3dd: Status 404 returned error can't find the container with id a7ccd7e2ab1ef8fad07f2ea1273a5eb0353c09c15f6883492f911a9a1b9ac3dd Nov 02 14:12:35.909234 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:35.909194 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" event={"ID":"641b4904-b6de-421c-b5a3-c6ab2a5fc006","Type":"ContainerStarted","Data":"421ee41457cf0ada493bb275b1faae2eabb7a30529bb373082b00cbb0dd1a12d"} Nov 02 14:12:35.909399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:35.909259 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" event={"ID":"641b4904-b6de-421c-b5a3-c6ab2a5fc006","Type":"ContainerStarted","Data":"a7ccd7e2ab1ef8fad07f2ea1273a5eb0353c09c15f6883492f911a9a1b9ac3dd"} Nov 02 14:12:35.966232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:35.966171 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" podStartSLOduration=1.966156565 podStartE2EDuration="1.966156565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:35.95254309 +0000 UTC m=+920.989648085" watchObservedRunningTime="2026-11-02 14:12:35.966156565 +0000 UTC m=+921.003261497" Nov 02 14:12:36.811921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:36.811893 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4"] Nov 02 14:12:36.815585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:36.815567 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:36.836669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:36.836643 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4"] Nov 02 14:12:36.916961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:36.916928 4431 generic.go:356] "Generic (PLEG): container finished" podID="641b4904-b6de-421c-b5a3-c6ab2a5fc006" containerID="421ee41457cf0ada493bb275b1faae2eabb7a30529bb373082b00cbb0dd1a12d" exitCode=0 Nov 02 14:12:36.917136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:36.917011 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" event={"ID":"641b4904-b6de-421c-b5a3-c6ab2a5fc006","Type":"ContainerDied","Data":"421ee41457cf0ada493bb275b1faae2eabb7a30529bb373082b00cbb0dd1a12d"} Nov 02 14:12:36.960334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:36.960292 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbv2m\" (UniqueName: \"kubernetes.io/projected/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-kube-api-access-qbv2m\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:36.960510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:36.960412 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:36.960510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:36.960466 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:37.061855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.061818 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:37.062321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.062031 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbv2m\" (UniqueName: \"kubernetes.io/projected/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-kube-api-access-qbv2m\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:37.062321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.062087 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:37.062685 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.062665 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:37.064274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.064255 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:37.087319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.087294 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbv2m\" (UniqueName: \"kubernetes.io/projected/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-kube-api-access-qbv2m\") pod \"maas-api-revoke-sub-8f1b84cb-h5qd4\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:37.129576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.129554 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:37.427868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.427838 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4"] Nov 02 14:12:37.429508 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:37.429477 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fa5579_2bd0_4be1_91b7_b7c71e00d214.slice/crio-8f1ceae80f7ce4db7dfbaf99bfa62a16b5f98d0ecc7050cbdb7cdf5e602d3f8c WatchSource:0}: Error finding container 8f1ceae80f7ce4db7dfbaf99bfa62a16b5f98d0ecc7050cbdb7cdf5e602d3f8c: Status 404 returned error can't find the container with id 8f1ceae80f7ce4db7dfbaf99bfa62a16b5f98d0ecc7050cbdb7cdf5e602d3f8c Nov 02 14:12:37.925172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.925065 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" event={"ID":"f7fa5579-2bd0-4be1-91b7-b7c71e00d214","Type":"ContainerStarted","Data":"2963455098e3a2233495cb5195194457da6ad3836043d1fa1457f2720e6c42c1"} Nov 02 14:12:37.925172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.925133 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" event={"ID":"f7fa5579-2bd0-4be1-91b7-b7c71e00d214","Type":"ContainerStarted","Data":"8f1ceae80f7ce4db7dfbaf99bfa62a16b5f98d0ecc7050cbdb7cdf5e602d3f8c"} Nov 02 14:12:37.985923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:37.985832 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" podStartSLOduration=1.985815969 podStartE2EDuration="1.985815969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:37.970423525 +0000 UTC m=+923.007528458" watchObservedRunningTime="2026-11-02 14:12:37.985815969 +0000 UTC m=+923.022920901" Nov 02 14:12:38.071097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.071060 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d"] Nov 02 14:12:38.072212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.072186 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:38.078039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.078018 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.111766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.111740 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d"] Nov 02 14:12:38.116056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.116032 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77"] Nov 02 14:12:38.116762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.116737 4431 cpu_manager.go:414] "containerMap: removing container" podUID="641b4904-b6de-421c-b5a3-c6ab2a5fc006" containerName="revoke-keys" Nov 02 14:12:38.116912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.116764 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641b4904-b6de-421c-b5a3-c6ab2a5fc006" containerName="revoke-keys" Nov 02 14:12:38.116994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.116936 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="641b4904-b6de-421c-b5a3-c6ab2a5fc006" containerName="revoke-keys" Nov 02 14:12:38.121789 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.121755 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.144516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.144493 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77"] Nov 02 14:12:38.176475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.176451 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-xks-ca\") pod \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " Nov 02 14:12:38.176629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.176543 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-openshift-ca\") pod \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " Nov 02 14:12:38.176629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.176606 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641b4904-b6de-421c-b5a3-c6ab2a5fc006-kube-api-access-ngwcw\" (UniqueName: \"kubernetes.io/projected/641b4904-b6de-421c-b5a3-c6ab2a5fc006-kube-api-access-ngwcw\") pod \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\" (UID: \"641b4904-b6de-421c-b5a3-c6ab2a5fc006\") " Nov 02 14:12:38.176939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.176911 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trhk\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-kube-api-access-7trhk\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.176939 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-openshift-ca" pod "641b4904-b6de-421c-b5a3-c6ab2a5fc006" (UID: "641b4904-b6de-421c-b5a3-c6ab2a5fc006"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:38.177231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177043 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59953df1-7268-4846-a4af-31361b224205-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177189 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177221 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59953df1-7268-4846-a4af-31361b224205-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177268 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177339 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177387 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177422 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177449 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.177639 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.177623 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:38.178852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.178823 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-xks-ca" pod "641b4904-b6de-421c-b5a3-c6ab2a5fc006" (UID: "641b4904-b6de-421c-b5a3-c6ab2a5fc006"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:38.178947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.178853 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641b4904-b6de-421c-b5a3-c6ab2a5fc006-kube-api-access-ngwcw" pod "641b4904-b6de-421c-b5a3-c6ab2a5fc006" (UID: "641b4904-b6de-421c-b5a3-c6ab2a5fc006"). InnerVolumeSpecName "kube-api-access-ngwcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:38.278319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278289 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.278467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278325 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.278467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278346 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5f7q\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-kube-api-access-l5f7q\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.278467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278373 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.278583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278487 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.278624 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278601 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.278685 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278669 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trhk\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-kube-api-access-7trhk\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.278739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278702 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.278812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278750 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.278812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278771 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.278912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278841 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.278912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278869 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59953df1-7268-4846-a4af-31361b224205-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278921 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278947 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59953df1-7268-4846-a4af-31361b224205-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278964 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.278985 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279006 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.279302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279033 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279065 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279134 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641b4904-b6de-421c-b5a3-c6ab2a5fc006-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:38.279302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279155 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngwcw\" (UniqueName: \"kubernetes.io/projected/641b4904-b6de-421c-b5a3-c6ab2a5fc006-kube-api-access-ngwcw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:38.279302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279253 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279534 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279431 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279534 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279478 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.279632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.279545 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59953df1-7268-4846-a4af-31361b224205-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.281321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.281296 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.281670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.281651 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59953df1-7268-4846-a4af-31361b224205-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.322694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.322667 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.323096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.323077 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trhk\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-kube-api-access-7trhk\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.379834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.379765 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.379941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.379867 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.379941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.379933 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.379972 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.379993 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5f7q\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-kube-api-access-l5f7q\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380023 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380123 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380183 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-certs\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380222 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380363 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380262 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-credential-socket\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380363 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380304 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380383 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380527 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-data\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.380693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.380674 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istiod-ca-cert\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.382285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.382260 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-envoy\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.382438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.382424 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-podinfo\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.394374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.394347 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:38.411694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.411668 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-token\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.411819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.411739 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5f7q\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-kube-api-access-l5f7q\") pod \"e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.441204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.441173 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:38.716007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.715982 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d"] Nov 02 14:12:38.716971 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:38.716945 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59953df1_7268_4846_a4af_31361b224205.slice/crio-1d893194881486747c240cab8e5b6794d970784279447ed0e7cf88909a7df9ae WatchSource:0}: Error finding container 1d893194881486747c240cab8e5b6794d970784279447ed0e7cf88909a7df9ae: Status 404 returned error can't find the container with id 1d893194881486747c240cab8e5b6794d970784279447ed0e7cf88909a7df9ae Nov 02 14:12:38.719535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.719506 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:38.719600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.719570 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:38.719600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.719596 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:38.759141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.759116 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77"] Nov 02 14:12:38.759407 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:38.759382 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c954aaf_4cb0_4b2b_86ed_77b79fd50c0d.slice/crio-258189f925880a2b613ef0ed62c74af1a674827d17f9fc5772e31e6c13d2074b WatchSource:0}: Error finding container 258189f925880a2b613ef0ed62c74af1a674827d17f9fc5772e31e6c13d2074b: Status 404 returned error can't find the container with id 258189f925880a2b613ef0ed62c74af1a674827d17f9fc5772e31e6c13d2074b Nov 02 14:12:38.761804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.761744 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:38.761905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.761830 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:38.761905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.761877 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:38.934221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.934194 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" Nov 02 14:12:38.934377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.934213 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4" event={"ID":"641b4904-b6de-421c-b5a3-c6ab2a5fc006","Type":"ContainerDied","Data":"a7ccd7e2ab1ef8fad07f2ea1273a5eb0353c09c15f6883492f911a9a1b9ac3dd"} Nov 02 14:12:38.934377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.934268 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7ccd7e2ab1ef8fad07f2ea1273a5eb0353c09c15f6883492f911a9a1b9ac3dd" Nov 02 14:12:38.935896 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.935865 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" event={"ID":"59953df1-7268-4846-a4af-31361b224205","Type":"ContainerStarted","Data":"106e7f7ec102bd39c9e8ac3355e5ba6080b1a9302a788fee182fe198212b00b6"} Nov 02 14:12:38.936036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.935917 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" event={"ID":"59953df1-7268-4846-a4af-31361b224205","Type":"ContainerStarted","Data":"1d893194881486747c240cab8e5b6794d970784279447ed0e7cf88909a7df9ae"} Nov 02 14:12:38.937563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.937542 4431 generic.go:356] "Generic (PLEG): container finished" podID="f7fa5579-2bd0-4be1-91b7-b7c71e00d214" containerID="2963455098e3a2233495cb5195194457da6ad3836043d1fa1457f2720e6c42c1" exitCode=0 Nov 02 14:12:38.937659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.937617 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" event={"ID":"f7fa5579-2bd0-4be1-91b7-b7c71e00d214","Type":"ContainerDied","Data":"2963455098e3a2233495cb5195194457da6ad3836043d1fa1457f2720e6c42c1"} Nov 02 14:12:38.939400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.939361 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" event={"ID":"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d","Type":"ContainerStarted","Data":"517d801f9611c704e27598d6d6acad4c219de62db7688119b14b0a7106bf87dc"} Nov 02 14:12:38.939515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.939403 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" event={"ID":"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d","Type":"ContainerStarted","Data":"258189f925880a2b613ef0ed62c74af1a674827d17f9fc5772e31e6c13d2074b"} Nov 02 14:12:38.999323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:38.999245 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podStartSLOduration=0.999233829 podStartE2EDuration="999.233829ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:38.998328727 +0000 UTC m=+924.035433660" watchObservedRunningTime="2026-11-02 14:12:38.999233829 +0000 UTC m=+924.036338761" Nov 02 14:12:39.107528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.107481 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podStartSLOduration=1.107467729 podStartE2EDuration="1.107467729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:39.106007223 +0000 UTC m=+924.143112155" watchObservedRunningTime="2026-11-02 14:12:39.107467729 +0000 UTC m=+924.144572660" Nov 02 14:12:39.394936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.394900 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:39.396572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.396538 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:12:39.396718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.396603 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 14:12:39.441859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.441832 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:39.443641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.443613 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 14:12:39.443737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.443657 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 14:12:39.446747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.446728 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925"] Nov 02 14:12:39.450835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.450816 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.498821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.498702 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925"] Nov 02 14:12:39.602796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.602428 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzc8\" (UniqueName: \"kubernetes.io/projected/1d1a733b-05ec-4aad-8ebc-f116c54580e4-kube-api-access-kxzc8\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.602796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.602527 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.602796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.602639 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.706854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.706457 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.706854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.706579 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzc8\" (UniqueName: \"kubernetes.io/projected/1d1a733b-05ec-4aad-8ebc-f116c54580e4-kube-api-access-kxzc8\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.706854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.706660 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.710803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.707545 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.714797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.713363 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.732563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.732530 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzc8\" (UniqueName: \"kubernetes.io/projected/1d1a733b-05ec-4aad-8ebc-f116c54580e4-kube-api-access-kxzc8\") pod \"maas-api-revoke-sub-80b8ef5d-2x925\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:39.764797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:39.764757 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:40.125707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.125670 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925"] Nov 02 14:12:40.130986 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:40.130940 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1a733b_05ec_4aad_8ebc_f116c54580e4.slice/crio-94a58d481011e519d13bc68e94b2330b8dfa936bca8dffb39e1ad309e4765b71 WatchSource:0}: Error finding container 94a58d481011e519d13bc68e94b2330b8dfa936bca8dffb39e1ad309e4765b71: Status 404 returned error can't find the container with id 94a58d481011e519d13bc68e94b2330b8dfa936bca8dffb39e1ad309e4765b71 Nov 02 14:12:40.234651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.234625 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:40.330371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.330263 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-openshift-ca\") pod \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " Nov 02 14:12:40.330547 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.330492 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-xks-ca\") pod \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " Nov 02 14:12:40.330635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.330559 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-kube-api-access-qbv2m\" (UniqueName: \"kubernetes.io/projected/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-kube-api-access-qbv2m\") pod \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\" (UID: \"f7fa5579-2bd0-4be1-91b7-b7c71e00d214\") " Nov 02 14:12:40.331510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.331482 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-openshift-ca" pod "f7fa5579-2bd0-4be1-91b7-b7c71e00d214" (UID: "f7fa5579-2bd0-4be1-91b7-b7c71e00d214"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:40.333754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.333734 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-xks-ca" pod "f7fa5579-2bd0-4be1-91b7-b7c71e00d214" (UID: "f7fa5579-2bd0-4be1-91b7-b7c71e00d214"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:40.333869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.333742 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-kube-api-access-qbv2m" pod "f7fa5579-2bd0-4be1-91b7-b7c71e00d214" (UID: "f7fa5579-2bd0-4be1-91b7-b7c71e00d214"). InnerVolumeSpecName "kube-api-access-qbv2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:40.419079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.418605 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:12:40.419079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.418668 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 14:12:40.432483 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.432439 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:40.432483 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.432486 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:40.432669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.432503 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbv2m\" (UniqueName: \"kubernetes.io/projected/f7fa5579-2bd0-4be1-91b7-b7c71e00d214-kube-api-access-qbv2m\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:40.442854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.442826 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 14:12:40.442987 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.442882 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 14:12:40.646805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.645556 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:12:40.955517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.955484 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" event={"ID":"1d1a733b-05ec-4aad-8ebc-f116c54580e4","Type":"ContainerStarted","Data":"5266c4843f63549eeb86eedd71c9f83a170c92aedf075d13ad28285a5205a7b3"} Nov 02 14:12:40.955716 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.955532 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" event={"ID":"1d1a733b-05ec-4aad-8ebc-f116c54580e4","Type":"ContainerStarted","Data":"94a58d481011e519d13bc68e94b2330b8dfa936bca8dffb39e1ad309e4765b71"} Nov 02 14:12:40.956953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.956911 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" event={"ID":"f7fa5579-2bd0-4be1-91b7-b7c71e00d214","Type":"ContainerDied","Data":"8f1ceae80f7ce4db7dfbaf99bfa62a16b5f98d0ecc7050cbdb7cdf5e602d3f8c"} Nov 02 14:12:40.956953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.956948 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4" Nov 02 14:12:40.957083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:40.956951 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f1ceae80f7ce4db7dfbaf99bfa62a16b5f98d0ecc7050cbdb7cdf5e602d3f8c" Nov 02 14:12:41.397802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:41.396877 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:12:41.397802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:41.396946 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 14:12:41.441809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:41.441765 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 14:12:41.442038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:41.442008 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 14:12:41.968330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:41.968296 4431 generic.go:356] "Generic (PLEG): container finished" podID="1d1a733b-05ec-4aad-8ebc-f116c54580e4" containerID="5266c4843f63549eeb86eedd71c9f83a170c92aedf075d13ad28285a5205a7b3" exitCode=0 Nov 02 14:12:41.968540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:41.968378 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" event={"ID":"1d1a733b-05ec-4aad-8ebc-f116c54580e4","Type":"ContainerDied","Data":"5266c4843f63549eeb86eedd71c9f83a170c92aedf075d13ad28285a5205a7b3"} Nov 02 14:12:42.406801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:42.399236 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:12:42.406801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:42.399330 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 14:12:42.442921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:42.442887 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 14:12:42.443173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:42.443145 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 14:12:43.444799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.444137 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:12:43.444799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.444240 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 14:12:43.445305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.444818 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 14:12:43.445305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.444879 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 14:12:43.887805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.878071 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:43.987518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.968444 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1a733b-05ec-4aad-8ebc-f116c54580e4-kube-api-access-kxzc8\" (UniqueName: \"kubernetes.io/projected/1d1a733b-05ec-4aad-8ebc-f116c54580e4-kube-api-access-kxzc8\") pod \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " Nov 02 14:12:43.987518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.968539 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-openshift-ca\") pod \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " Nov 02 14:12:43.987518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.968740 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-xks-ca\") pod \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\" (UID: \"1d1a733b-05ec-4aad-8ebc-f116c54580e4\") " Nov 02 14:12:43.999484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.998301 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-openshift-ca" pod "1d1a733b-05ec-4aad-8ebc-f116c54580e4" (UID: "1d1a733b-05ec-4aad-8ebc-f116c54580e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:43.999484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:43.999441 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1a733b-05ec-4aad-8ebc-f116c54580e4-kube-api-access-kxzc8" pod "1d1a733b-05ec-4aad-8ebc-f116c54580e4" (UID: "1d1a733b-05ec-4aad-8ebc-f116c54580e4"). InnerVolumeSpecName "kube-api-access-kxzc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:44.112802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.073276 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-xks-ca" pod "1d1a733b-05ec-4aad-8ebc-f116c54580e4" (UID: "1d1a733b-05ec-4aad-8ebc-f116c54580e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:44.131806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.129912 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzc8\" (UniqueName: \"kubernetes.io/projected/1d1a733b-05ec-4aad-8ebc-f116c54580e4-kube-api-access-kxzc8\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:44.131806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.129945 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:44.131806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.129960 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1a733b-05ec-4aad-8ebc-f116c54580e4-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:44.136645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.134979 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" event={"ID":"1d1a733b-05ec-4aad-8ebc-f116c54580e4","Type":"ContainerDied","Data":"94a58d481011e519d13bc68e94b2330b8dfa936bca8dffb39e1ad309e4765b71"} Nov 02 14:12:44.136645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.135035 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94a58d481011e519d13bc68e94b2330b8dfa936bca8dffb39e1ad309e4765b71" Nov 02 14:12:44.136645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.135110 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925" Nov 02 14:12:44.401408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.401293 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:12:44.401669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.401625 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 14:12:44.442616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.442590 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 14:12:44.442900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.442858 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 14:12:44.531894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.528839 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v"] Nov 02 14:12:44.539094 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.539064 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f7fa5579-2bd0-4be1-91b7-b7c71e00d214" containerName="revoke-keys" Nov 02 14:12:44.539282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.539267 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fa5579-2bd0-4be1-91b7-b7c71e00d214" containerName="revoke-keys" Nov 02 14:12:44.539398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.539386 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1d1a733b-05ec-4aad-8ebc-f116c54580e4" containerName="revoke-keys" Nov 02 14:12:44.539496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.539485 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1a733b-05ec-4aad-8ebc-f116c54580e4" containerName="revoke-keys" Nov 02 14:12:44.539770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.539756 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fa5579-2bd0-4be1-91b7-b7c71e00d214" containerName="revoke-keys" Nov 02 14:12:44.539918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.539906 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1a733b-05ec-4aad-8ebc-f116c54580e4" containerName="revoke-keys" Nov 02 14:12:44.544307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.544282 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v"] Nov 02 14:12:44.544450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.544439 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.561113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.561088 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:12:44.561207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.561137 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:12:44.643957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.643920 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.644141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.643997 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.644141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.644097 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb26c\" (UniqueName: \"kubernetes.io/projected/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-kube-api-access-bb26c\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.745566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.745474 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.745566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.745542 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.745818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.745619 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb26c\" (UniqueName: \"kubernetes.io/projected/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-kube-api-access-bb26c\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.746128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.746108 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.747839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.747812 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.822675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.821532 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb26c\" (UniqueName: \"kubernetes.io/projected/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-kube-api-access-bb26c\") pod \"maas-api-revoke-sub-2baee653-t8l9v\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:44.858196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:44.858170 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:45.214703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:45.214096 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v"] Nov 02 14:12:45.216835 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:45.216808 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db8ce96_a4c2_4be2_ab06_82ebfaef2708.slice/crio-1c0c65e0127e2c90b4240de5fc36a34aa3e26aa4258731e866736934de787032 WatchSource:0}: Error finding container 1c0c65e0127e2c90b4240de5fc36a34aa3e26aa4258731e866736934de787032: Status 404 returned error can't find the container with id 1c0c65e0127e2c90b4240de5fc36a34aa3e26aa4258731e866736934de787032 Nov 02 14:12:45.395806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:45.395106 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 14:12:45.395806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:45.395164 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 14:12:45.441670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:45.441647 4431 patch_prober.go:29] interesting pod/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 14:12:45.441830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:45.441688 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 14:12:46.157148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.157114 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" event={"ID":"1db8ce96-a4c2-4be2-ab06-82ebfaef2708","Type":"ContainerStarted","Data":"15ace6f857a85ac96cbb71e5c6337f0ffe2b90f04fe100805718049b38681a63"} Nov 02 14:12:46.157521 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.157163 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" event={"ID":"1db8ce96-a4c2-4be2-ab06-82ebfaef2708","Type":"ContainerStarted","Data":"1c0c65e0127e2c90b4240de5fc36a34aa3e26aa4258731e866736934de787032"} Nov 02 14:12:46.203282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.203241 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" podStartSLOduration=2.203227143 podStartE2EDuration="2.203227143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:46.201542789 +0000 UTC m=+931.238647739" watchObservedRunningTime="2026-11-02 14:12:46.203227143 +0000 UTC m=+931.240332121" Nov 02 14:12:46.399133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.399102 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:46.399603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.399586 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:46.400212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.400195 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:12:46.446214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.446157 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:46.446453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.446428 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:46.447255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:46.447221 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:12:47.165417 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:47.165380 4431 generic.go:356] "Generic (PLEG): container finished" podID="1db8ce96-a4c2-4be2-ab06-82ebfaef2708" containerID="15ace6f857a85ac96cbb71e5c6337f0ffe2b90f04fe100805718049b38681a63" exitCode=0 Nov 02 14:12:47.165836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:47.165426 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" event={"ID":"1db8ce96-a4c2-4be2-ab06-82ebfaef2708","Type":"ContainerDied","Data":"15ace6f857a85ac96cbb71e5c6337f0ffe2b90f04fe100805718049b38681a63"} Nov 02 14:12:48.325207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.325181 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:48.385488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.385457 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-xks-ca\") pod \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " Nov 02 14:12:48.385642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.385501 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-kube-api-access-bb26c\" (UniqueName: \"kubernetes.io/projected/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-kube-api-access-bb26c\") pod \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " Nov 02 14:12:48.385703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.385650 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-openshift-ca\") pod \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\" (UID: \"1db8ce96-a4c2-4be2-ab06-82ebfaef2708\") " Nov 02 14:12:48.386105 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.386071 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-openshift-ca" pod "1db8ce96-a4c2-4be2-ab06-82ebfaef2708" (UID: "1db8ce96-a4c2-4be2-ab06-82ebfaef2708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:48.387461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.387441 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-xks-ca" pod "1db8ce96-a4c2-4be2-ab06-82ebfaef2708" (UID: "1db8ce96-a4c2-4be2-ab06-82ebfaef2708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:48.387543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.387527 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-kube-api-access-bb26c" pod "1db8ce96-a4c2-4be2-ab06-82ebfaef2708" (UID: "1db8ce96-a4c2-4be2-ab06-82ebfaef2708"). InnerVolumeSpecName "kube-api-access-bb26c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:48.487038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.486983 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:48.487038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.487004 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:48.487038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:48.487015 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb26c\" (UniqueName: \"kubernetes.io/projected/1db8ce96-a4c2-4be2-ab06-82ebfaef2708-kube-api-access-bb26c\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:49.178428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:49.178399 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" event={"ID":"1db8ce96-a4c2-4be2-ab06-82ebfaef2708","Type":"ContainerDied","Data":"1c0c65e0127e2c90b4240de5fc36a34aa3e26aa4258731e866736934de787032"} Nov 02 14:12:49.178428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:49.178424 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v" Nov 02 14:12:49.178626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:49.178441 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c0c65e0127e2c90b4240de5fc36a34aa3e26aa4258731e866736934de787032" Nov 02 14:12:51.283635 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:12:51.283602 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:12:54.902049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.902015 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm"] Nov 02 14:12:54.902506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.902494 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1db8ce96-a4c2-4be2-ab06-82ebfaef2708" containerName="revoke-keys" Nov 02 14:12:54.902553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.902507 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db8ce96-a4c2-4be2-ab06-82ebfaef2708" containerName="revoke-keys" Nov 02 14:12:54.902619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.902610 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db8ce96-a4c2-4be2-ab06-82ebfaef2708" containerName="revoke-keys" Nov 02 14:12:54.907191 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.907172 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:54.926053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.926028 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm"] Nov 02 14:12:54.935030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.935005 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:12:54.935140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.935125 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:12:54.956897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.956867 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:54.957015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.956901 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:54.957084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:54.957064 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngsw\" (UniqueName: \"kubernetes.io/projected/93ea4292-a174-412e-9777-e33b9d197ac5-kube-api-access-6ngsw\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:55.058065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.058036 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngsw\" (UniqueName: \"kubernetes.io/projected/93ea4292-a174-412e-9777-e33b9d197ac5-kube-api-access-6ngsw\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:55.058184 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.058086 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:55.058184 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.058107 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:55.058745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.058714 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:55.060460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.060435 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:55.083185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.083159 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngsw\" (UniqueName: \"kubernetes.io/projected/93ea4292-a174-412e-9777-e33b9d197ac5-kube-api-access-6ngsw\") pod \"maas-api-revoke-sub-62ede89e-n4kqm\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:55.220299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.220240 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:55.532998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.524917 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm"] Nov 02 14:12:55.556372 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:55.545075 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ea4292_a174_412e_9777_e33b9d197ac5.slice/crio-d46f36d08e0deaa2d9a4b393b3475d6fbb106b6a48223dc86185e5627a56b5bc WatchSource:0}: Error finding container d46f36d08e0deaa2d9a4b393b3475d6fbb106b6a48223dc86185e5627a56b5bc: Status 404 returned error can't find the container with id d46f36d08e0deaa2d9a4b393b3475d6fbb106b6a48223dc86185e5627a56b5bc Nov 02 14:12:55.595869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.595838 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm"] Nov 02 14:12:55.602218 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.602192 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.621341 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.621302 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-de8cc7-openshift-default-dockercfg-8bqn4\"" Nov 02 14:12:55.673922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.673868 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm"] Nov 02 14:12:55.675008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.674983 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-socket\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.675241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.675222 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-credential-socket\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.675373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.675348 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-data\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.675559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.675543 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273ac84f-f3ab-44e7-998b-4fa269667ca0-istiod-ca-cert\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.675681 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.675667 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-token\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.675864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.675848 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-podinfo\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.676035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.676019 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnww\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-kube-api-access-swnww\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.676186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.676168 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-envoy\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.676365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.676349 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-certs\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.782834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.781643 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-envoy\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.782834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.781741 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-certs\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.782834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.781794 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-socket\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.782834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.781847 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-credential-socket\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.782834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.781871 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-data\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.782834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.781966 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273ac84f-f3ab-44e7-998b-4fa269667ca0-istiod-ca-cert\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.791868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.783384 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-socket\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.791868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.791439 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-credential-socket\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.791868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.791679 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-data\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.793555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.792317 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273ac84f-f3ab-44e7-998b-4fa269667ca0-istiod-ca-cert\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.793555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.782000 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-token\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.793555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.792411 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-podinfo\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.793555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.792482 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnww\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-kube-api-access-swnww\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.813797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.810153 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-certs\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.878799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.877902 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-envoy\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.881588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.879327 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-token\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.881588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.880071 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnww\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-kube-api-access-swnww\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.881588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.880532 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-podinfo\") pod \"e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:55.926881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:55.926549 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:56.397801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:56.395278 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" event={"ID":"93ea4292-a174-412e-9777-e33b9d197ac5","Type":"ContainerStarted","Data":"4e4ea777f2155e44d63e92eaee2c708971e84623e3fb7726a7aa63e76b04b233"} Nov 02 14:12:56.397801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:56.395416 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" event={"ID":"93ea4292-a174-412e-9777-e33b9d197ac5","Type":"ContainerStarted","Data":"d46f36d08e0deaa2d9a4b393b3475d6fbb106b6a48223dc86185e5627a56b5bc"} Nov 02 14:12:56.457101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:56.457038 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" podStartSLOduration=2.457019218 podStartE2EDuration="2.457019218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:56.456046731 +0000 UTC m=+941.493151688" watchObservedRunningTime="2026-11-02 14:12:56.457019218 +0000 UTC m=+941.494124152" Nov 02 14:12:57.254994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:57.233167 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm"] Nov 02 14:12:57.334186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:57.321610 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:57.334186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:57.321708 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:57.334186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:57.321759 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:12:57.443857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:57.437979 4431 generic.go:356] "Generic (PLEG): container finished" podID="93ea4292-a174-412e-9777-e33b9d197ac5" containerID="4e4ea777f2155e44d63e92eaee2c708971e84623e3fb7726a7aa63e76b04b233" exitCode=0 Nov 02 14:12:57.443857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:57.438235 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" event={"ID":"93ea4292-a174-412e-9777-e33b9d197ac5","Type":"ContainerDied","Data":"4e4ea777f2155e44d63e92eaee2c708971e84623e3fb7726a7aa63e76b04b233"} Nov 02 14:12:57.443857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:57.441049 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" event={"ID":"273ac84f-f3ab-44e7-998b-4fa269667ca0","Type":"ContainerStarted","Data":"c4f40fe06a47468d136dcc45a12f4eed979d187bb9abeb6fe96ee7c5fa46f3c4"} Nov 02 14:12:58.018289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.018197 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x"] Nov 02 14:12:58.034878 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.034841 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.035180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.035147 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x"] Nov 02 14:12:58.055915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.055886 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.056037 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.055959 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2clt\" (UniqueName: \"kubernetes.io/projected/b428bd72-4fb6-4a99-96dc-b95287d36e8b-kube-api-access-h2clt\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.056135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.056110 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.156867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.156824 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.157058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.156898 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2clt\" (UniqueName: \"kubernetes.io/projected/b428bd72-4fb6-4a99-96dc-b95287d36e8b-kube-api-access-h2clt\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.157058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.156935 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.157573 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.157549 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.159500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.159472 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.182128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.182101 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2clt\" (UniqueName: \"kubernetes.io/projected/b428bd72-4fb6-4a99-96dc-b95287d36e8b-kube-api-access-h2clt\") pod \"maas-api-revoke-sub-249e699c-2cf4x\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.365643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.365561 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:12:58.450744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.450711 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" event={"ID":"273ac84f-f3ab-44e7-998b-4fa269667ca0","Type":"ContainerStarted","Data":"3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b"} Nov 02 14:12:58.526725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.526666 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" podStartSLOduration=3.526651904 podStartE2EDuration="3.526651904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:58.512527583 +0000 UTC m=+943.549632528" watchObservedRunningTime="2026-11-02 14:12:58.526651904 +0000 UTC m=+943.563756835" Nov 02 14:12:58.592092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.592067 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:58.661569 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.661534 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-xks-ca\") pod \"93ea4292-a174-412e-9777-e33b9d197ac5\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " Nov 02 14:12:58.661747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.661583 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ea4292-a174-412e-9777-e33b9d197ac5-kube-api-access-6ngsw\" (UniqueName: \"kubernetes.io/projected/93ea4292-a174-412e-9777-e33b9d197ac5-kube-api-access-6ngsw\") pod \"93ea4292-a174-412e-9777-e33b9d197ac5\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " Nov 02 14:12:58.661747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.661653 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-openshift-ca\") pod \"93ea4292-a174-412e-9777-e33b9d197ac5\" (UID: \"93ea4292-a174-412e-9777-e33b9d197ac5\") " Nov 02 14:12:58.662555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.662499 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-openshift-ca" pod "93ea4292-a174-412e-9777-e33b9d197ac5" (UID: "93ea4292-a174-412e-9777-e33b9d197ac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:12:58.664999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.664975 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-xks-ca" pod "93ea4292-a174-412e-9777-e33b9d197ac5" (UID: "93ea4292-a174-412e-9777-e33b9d197ac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:12:58.669373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.669310 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ea4292-a174-412e-9777-e33b9d197ac5-kube-api-access-6ngsw" pod "93ea4292-a174-412e-9777-e33b9d197ac5" (UID: "93ea4292-a174-412e-9777-e33b9d197ac5"). InnerVolumeSpecName "kube-api-access-6ngsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:12:58.675558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.675534 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x"] Nov 02 14:12:58.676764 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:12:58.676742 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb428bd72_4fb6_4a99_96dc_b95287d36e8b.slice/crio-ddf0c39da890abedb0d1bbd3c3d5576857aec4013e36ceb8f68be30a877e7cf0 WatchSource:0}: Error finding container ddf0c39da890abedb0d1bbd3c3d5576857aec4013e36ceb8f68be30a877e7cf0: Status 404 returned error can't find the container with id ddf0c39da890abedb0d1bbd3c3d5576857aec4013e36ceb8f68be30a877e7cf0 Nov 02 14:12:58.763410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.763380 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:58.763410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.763408 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ea4292-a174-412e-9777-e33b9d197ac5-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:58.763410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.763417 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngsw\" (UniqueName: \"kubernetes.io/projected/93ea4292-a174-412e-9777-e33b9d197ac5-kube-api-access-6ngsw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:12:58.927355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.927286 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:58.932786 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:58.932749 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:59.458921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:59.458887 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" event={"ID":"b428bd72-4fb6-4a99-96dc-b95287d36e8b","Type":"ContainerStarted","Data":"8c85bf496df3911600f39eba8afcdeb2a7faf97c4b865b77e3a9628eed942834"} Nov 02 14:12:59.459363 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:59.458978 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" event={"ID":"b428bd72-4fb6-4a99-96dc-b95287d36e8b","Type":"ContainerStarted","Data":"ddf0c39da890abedb0d1bbd3c3d5576857aec4013e36ceb8f68be30a877e7cf0"} Nov 02 14:12:59.460223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:59.460202 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" Nov 02 14:12:59.460325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:59.460229 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm" event={"ID":"93ea4292-a174-412e-9777-e33b9d197ac5","Type":"ContainerDied","Data":"d46f36d08e0deaa2d9a4b393b3475d6fbb106b6a48223dc86185e5627a56b5bc"} Nov 02 14:12:59.460325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:59.460270 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46f36d08e0deaa2d9a4b393b3475d6fbb106b6a48223dc86185e5627a56b5bc" Nov 02 14:12:59.460625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:59.460602 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:59.461602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:59.461583 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:12:59.509099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:12:59.509054 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" podStartSLOduration=2.509041607 podStartE2EDuration="2.509041607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:59.506937696 +0000 UTC m=+944.544042627" watchObservedRunningTime="2026-11-02 14:12:59.509041607 +0000 UTC m=+944.546146539" Nov 02 14:13:00.468307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:00.468271 4431 generic.go:356] "Generic (PLEG): container finished" podID="b428bd72-4fb6-4a99-96dc-b95287d36e8b" containerID="8c85bf496df3911600f39eba8afcdeb2a7faf97c4b865b77e3a9628eed942834" exitCode=0 Nov 02 14:13:00.468705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:00.468356 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" event={"ID":"b428bd72-4fb6-4a99-96dc-b95287d36e8b","Type":"ContainerDied","Data":"8c85bf496df3911600f39eba8afcdeb2a7faf97c4b865b77e3a9628eed942834"} Nov 02 14:13:01.625388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.625359 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:13:01.695951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.695918 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-openshift-ca\") pod \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " Nov 02 14:13:01.696126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.695993 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b428bd72-4fb6-4a99-96dc-b95287d36e8b-kube-api-access-h2clt\" (UniqueName: \"kubernetes.io/projected/b428bd72-4fb6-4a99-96dc-b95287d36e8b-kube-api-access-h2clt\") pod \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " Nov 02 14:13:01.696126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.696043 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-xks-ca\") pod \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\" (UID: \"b428bd72-4fb6-4a99-96dc-b95287d36e8b\") " Nov 02 14:13:01.696315 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.696293 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-openshift-ca" pod "b428bd72-4fb6-4a99-96dc-b95287d36e8b" (UID: "b428bd72-4fb6-4a99-96dc-b95287d36e8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:01.696543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.696524 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:01.698292 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.698262 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-xks-ca" pod "b428bd72-4fb6-4a99-96dc-b95287d36e8b" (UID: "b428bd72-4fb6-4a99-96dc-b95287d36e8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:01.698402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.698382 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b428bd72-4fb6-4a99-96dc-b95287d36e8b-kube-api-access-h2clt" pod "b428bd72-4fb6-4a99-96dc-b95287d36e8b" (UID: "b428bd72-4fb6-4a99-96dc-b95287d36e8b"). InnerVolumeSpecName "kube-api-access-h2clt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:01.797343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.797264 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2clt\" (UniqueName: \"kubernetes.io/projected/b428bd72-4fb6-4a99-96dc-b95287d36e8b-kube-api-access-h2clt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:01.797343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:01.797295 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b428bd72-4fb6-4a99-96dc-b95287d36e8b-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:02.483967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:02.483939 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" Nov 02 14:13:02.484113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:02.483980 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x" event={"ID":"b428bd72-4fb6-4a99-96dc-b95287d36e8b","Type":"ContainerDied","Data":"ddf0c39da890abedb0d1bbd3c3d5576857aec4013e36ceb8f68be30a877e7cf0"} Nov 02 14:13:02.484113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:02.484025 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddf0c39da890abedb0d1bbd3c3d5576857aec4013e36ceb8f68be30a877e7cf0" Nov 02 14:13:04.324251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.324209 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z"] Nov 02 14:13:04.324740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.324727 4431 cpu_manager.go:414] "containerMap: removing container" podUID="93ea4292-a174-412e-9777-e33b9d197ac5" containerName="revoke-keys" Nov 02 14:13:04.324819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.324742 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ea4292-a174-412e-9777-e33b9d197ac5" containerName="revoke-keys" Nov 02 14:13:04.324819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.324755 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b428bd72-4fb6-4a99-96dc-b95287d36e8b" containerName="revoke-keys" Nov 02 14:13:04.324819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.324760 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b428bd72-4fb6-4a99-96dc-b95287d36e8b" containerName="revoke-keys" Nov 02 14:13:04.324917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.324875 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ea4292-a174-412e-9777-e33b9d197ac5" containerName="revoke-keys" Nov 02 14:13:04.324917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.324886 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b428bd72-4fb6-4a99-96dc-b95287d36e8b" containerName="revoke-keys" Nov 02 14:13:04.328420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.328404 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.357456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.357425 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:13:04.357604 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.357531 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:13:04.361288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.361266 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z"] Nov 02 14:13:04.428823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.428787 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.429000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.428959 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzct\" (UniqueName: \"kubernetes.io/projected/1c3366b2-3851-4a59-b1e7-9af8b7c70602-kube-api-access-zwzct\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.429051 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.429011 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.529904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.529871 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.530072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.529967 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzct\" (UniqueName: \"kubernetes.io/projected/1c3366b2-3851-4a59-b1e7-9af8b7c70602-kube-api-access-zwzct\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.530072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.530007 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.530553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.530529 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.532388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.532360 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.554724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.554698 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzct\" (UniqueName: \"kubernetes.io/projected/1c3366b2-3851-4a59-b1e7-9af8b7c70602-kube-api-access-zwzct\") pod \"maas-api-revoke-sub-f3e2927b-dhh6z\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.642909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.642887 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:04.943487 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:13:04.943457 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3366b2_3851_4a59_b1e7_9af8b7c70602.slice/crio-7a4350372df5ed47edb068724a4ada6b763febd4775a1ae7f0b0f60b785e1c99 WatchSource:0}: Error finding container 7a4350372df5ed47edb068724a4ada6b763febd4775a1ae7f0b0f60b785e1c99: Status 404 returned error can't find the container with id 7a4350372df5ed47edb068724a4ada6b763febd4775a1ae7f0b0f60b785e1c99 Nov 02 14:13:04.944156 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:04.944131 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z"] Nov 02 14:13:05.503641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:05.503608 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" event={"ID":"1c3366b2-3851-4a59-b1e7-9af8b7c70602","Type":"ContainerStarted","Data":"13b641a0a836eef23ab43bff96e4cc33934376722ef47d32505cbd50af79c539"} Nov 02 14:13:05.504151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:05.503667 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" event={"ID":"1c3366b2-3851-4a59-b1e7-9af8b7c70602","Type":"ContainerStarted","Data":"7a4350372df5ed47edb068724a4ada6b763febd4775a1ae7f0b0f60b785e1c99"} Nov 02 14:13:05.549756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:05.549673 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" podStartSLOduration=1.549659814 podStartE2EDuration="1.549659814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:05.54631919 +0000 UTC m=+950.583424121" watchObservedRunningTime="2026-11-02 14:13:05.549659814 +0000 UTC m=+950.586764746" Nov 02 14:13:05.866735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:05.866700 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw"] Nov 02 14:13:06.423819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.423768 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6"] Nov 02 14:13:06.427946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.427927 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.448152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.448125 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6"] Nov 02 14:13:06.510887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.510852 4431 generic.go:356] "Generic (PLEG): container finished" podID="1c3366b2-3851-4a59-b1e7-9af8b7c70602" containerID="13b641a0a836eef23ab43bff96e4cc33934376722ef47d32505cbd50af79c539" exitCode=0 Nov 02 14:13:06.511291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.510938 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" event={"ID":"1c3366b2-3851-4a59-b1e7-9af8b7c70602","Type":"ContainerDied","Data":"13b641a0a836eef23ab43bff96e4cc33934376722ef47d32505cbd50af79c539"} Nov 02 14:13:06.562905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.562872 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.563065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.563042 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t299f\" (UniqueName: \"kubernetes.io/projected/be2339e3-6752-4dc0-ab8f-144f260b3a1a-kube-api-access-t299f\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.563200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.563185 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.664438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.664404 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t299f\" (UniqueName: \"kubernetes.io/projected/be2339e3-6752-4dc0-ab8f-144f260b3a1a-kube-api-access-t299f\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.664592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.664514 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.664592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.664567 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.665144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.665119 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.666707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.666688 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.689191 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.689135 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t299f\" (UniqueName: \"kubernetes.io/projected/be2339e3-6752-4dc0-ab8f-144f260b3a1a-kube-api-access-t299f\") pod \"maas-api-revoke-sub-eb68d371-65kh6\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:06.745860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:06.745837 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:07.049235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.049206 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6"] Nov 02 14:13:07.049528 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:13:07.049508 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2339e3_6752_4dc0_ab8f_144f260b3a1a.slice/crio-746add1084b43d052b9bab8540743c817496a1b46477149376b8f56bad64a57e WatchSource:0}: Error finding container 746add1084b43d052b9bab8540743c817496a1b46477149376b8f56bad64a57e: Status 404 returned error can't find the container with id 746add1084b43d052b9bab8540743c817496a1b46477149376b8f56bad64a57e Nov 02 14:13:07.520263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.520228 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" event={"ID":"be2339e3-6752-4dc0-ab8f-144f260b3a1a","Type":"ContainerStarted","Data":"766bc3c3c4ea09d314f4f7f545c5d8daa24754dd862452088f34ab36c2bb5d11"} Nov 02 14:13:07.520755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.520287 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" event={"ID":"be2339e3-6752-4dc0-ab8f-144f260b3a1a","Type":"ContainerStarted","Data":"746add1084b43d052b9bab8540743c817496a1b46477149376b8f56bad64a57e"} Nov 02 14:13:07.570978 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.570924 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" podStartSLOduration=1.5709091499999999 podStartE2EDuration="1.57090915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:07.569554558 +0000 UTC m=+952.606659490" watchObservedRunningTime="2026-11-02 14:13:07.57090915 +0000 UTC m=+952.608014082" Nov 02 14:13:07.725798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.724565 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:07.780630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.780592 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-xks-ca\") pod \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " Nov 02 14:13:07.780849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.780738 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3366b2-3851-4a59-b1e7-9af8b7c70602-kube-api-access-zwzct\" (UniqueName: \"kubernetes.io/projected/1c3366b2-3851-4a59-b1e7-9af8b7c70602-kube-api-access-zwzct\") pod \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " Nov 02 14:13:07.781165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.781137 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-openshift-ca\") pod \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\" (UID: \"1c3366b2-3851-4a59-b1e7-9af8b7c70602\") " Nov 02 14:13:07.793942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.781968 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-openshift-ca" pod "1c3366b2-3851-4a59-b1e7-9af8b7c70602" (UID: "1c3366b2-3851-4a59-b1e7-9af8b7c70602"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:07.794922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.794894 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-xks-ca" pod "1c3366b2-3851-4a59-b1e7-9af8b7c70602" (UID: "1c3366b2-3851-4a59-b1e7-9af8b7c70602"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:07.797277 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.796969 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3366b2-3851-4a59-b1e7-9af8b7c70602-kube-api-access-zwzct" pod "1c3366b2-3851-4a59-b1e7-9af8b7c70602" (UID: "1c3366b2-3851-4a59-b1e7-9af8b7c70602"). InnerVolumeSpecName "kube-api-access-zwzct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:07.882939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.882901 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:07.882939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.882932 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwzct\" (UniqueName: \"kubernetes.io/projected/1c3366b2-3851-4a59-b1e7-9af8b7c70602-kube-api-access-zwzct\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:07.883127 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:07.882946 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3366b2-3851-4a59-b1e7-9af8b7c70602-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:08.528144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:08.528116 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" Nov 02 14:13:08.528576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:08.528115 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z" event={"ID":"1c3366b2-3851-4a59-b1e7-9af8b7c70602","Type":"ContainerDied","Data":"7a4350372df5ed47edb068724a4ada6b763febd4775a1ae7f0b0f60b785e1c99"} Nov 02 14:13:08.528576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:08.528201 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a4350372df5ed47edb068724a4ada6b763febd4775a1ae7f0b0f60b785e1c99" Nov 02 14:13:08.530012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:08.529985 4431 generic.go:356] "Generic (PLEG): container finished" podID="be2339e3-6752-4dc0-ab8f-144f260b3a1a" containerID="766bc3c3c4ea09d314f4f7f545c5d8daa24754dd862452088f34ab36c2bb5d11" exitCode=0 Nov 02 14:13:08.530159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:08.530070 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" event={"ID":"be2339e3-6752-4dc0-ab8f-144f260b3a1a","Type":"ContainerDied","Data":"766bc3c3c4ea09d314f4f7f545c5d8daa24754dd862452088f34ab36c2bb5d11"} Nov 02 14:13:09.686599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.686578 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:09.805990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.805921 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2339e3-6752-4dc0-ab8f-144f260b3a1a-kube-api-access-t299f\" (UniqueName: \"kubernetes.io/projected/be2339e3-6752-4dc0-ab8f-144f260b3a1a-kube-api-access-t299f\") pod \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " Nov 02 14:13:09.806132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.805991 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-xks-ca\") pod \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " Nov 02 14:13:09.806132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.806076 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-openshift-ca\") pod \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\" (UID: \"be2339e3-6752-4dc0-ab8f-144f260b3a1a\") " Nov 02 14:13:09.806541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.806515 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-openshift-ca" pod "be2339e3-6752-4dc0-ab8f-144f260b3a1a" (UID: "be2339e3-6752-4dc0-ab8f-144f260b3a1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:09.808151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.808127 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2339e3-6752-4dc0-ab8f-144f260b3a1a-kube-api-access-t299f" pod "be2339e3-6752-4dc0-ab8f-144f260b3a1a" (UID: "be2339e3-6752-4dc0-ab8f-144f260b3a1a"). InnerVolumeSpecName "kube-api-access-t299f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:09.808454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.808434 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-xks-ca" pod "be2339e3-6752-4dc0-ab8f-144f260b3a1a" (UID: "be2339e3-6752-4dc0-ab8f-144f260b3a1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:09.907288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.907261 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t299f\" (UniqueName: \"kubernetes.io/projected/be2339e3-6752-4dc0-ab8f-144f260b3a1a-kube-api-access-t299f\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:09.907288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.907287 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:09.907422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:09.907297 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2339e3-6752-4dc0-ab8f-144f260b3a1a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:10.363964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.363934 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz"] Nov 02 14:13:10.364475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.364460 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1c3366b2-3851-4a59-b1e7-9af8b7c70602" containerName="revoke-keys" Nov 02 14:13:10.364527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.364476 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3366b2-3851-4a59-b1e7-9af8b7c70602" containerName="revoke-keys" Nov 02 14:13:10.364527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.364490 4431 cpu_manager.go:414] "containerMap: removing container" podUID="be2339e3-6752-4dc0-ab8f-144f260b3a1a" containerName="revoke-keys" Nov 02 14:13:10.364527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.364495 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2339e3-6752-4dc0-ab8f-144f260b3a1a" containerName="revoke-keys" Nov 02 14:13:10.364614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.364583 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3366b2-3851-4a59-b1e7-9af8b7c70602" containerName="revoke-keys" Nov 02 14:13:10.364614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.364593 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2339e3-6752-4dc0-ab8f-144f260b3a1a" containerName="revoke-keys" Nov 02 14:13:10.369199 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.369177 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.390806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.390757 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz"] Nov 02 14:13:10.514007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.513970 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.514197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.514101 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.514197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.514132 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jszsm\" (UniqueName: \"kubernetes.io/projected/1f766458-1e64-452f-84f6-e2ff95fdd36a-kube-api-access-jszsm\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.546255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.546224 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" Nov 02 14:13:10.546255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.546248 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6" event={"ID":"be2339e3-6752-4dc0-ab8f-144f260b3a1a","Type":"ContainerDied","Data":"746add1084b43d052b9bab8540743c817496a1b46477149376b8f56bad64a57e"} Nov 02 14:13:10.546440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.546294 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="746add1084b43d052b9bab8540743c817496a1b46477149376b8f56bad64a57e" Nov 02 14:13:10.615500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.615416 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.615662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.615504 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.615662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.615532 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jszsm\" (UniqueName: \"kubernetes.io/projected/1f766458-1e64-452f-84f6-e2ff95fdd36a-kube-api-access-jszsm\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.616284 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.616264 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.617819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.617771 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.640375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.640343 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jszsm\" (UniqueName: \"kubernetes.io/projected/1f766458-1e64-452f-84f6-e2ff95fdd36a-kube-api-access-jszsm\") pod \"maas-api-revoke-sub-521ff25e-p79wz\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:10.703571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:10.703543 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:11.005319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:11.005287 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz"] Nov 02 14:13:11.007959 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:13:11.007924 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f766458_1e64_452f_84f6_e2ff95fdd36a.slice/crio-9f4bc1f6bd4cd03a5e3dafad41c397063f9266c1b24fe5e53c14a08953f3154e WatchSource:0}: Error finding container 9f4bc1f6bd4cd03a5e3dafad41c397063f9266c1b24fe5e53c14a08953f3154e: Status 404 returned error can't find the container with id 9f4bc1f6bd4cd03a5e3dafad41c397063f9266c1b24fe5e53c14a08953f3154e Nov 02 14:13:11.554296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:11.554219 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" event={"ID":"1f766458-1e64-452f-84f6-e2ff95fdd36a","Type":"ContainerStarted","Data":"d16bec4b9ef098e21063544b393670d9381b831b52cd83c6148d753da0f5ac26"} Nov 02 14:13:11.554296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:11.554270 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" event={"ID":"1f766458-1e64-452f-84f6-e2ff95fdd36a","Type":"ContainerStarted","Data":"9f4bc1f6bd4cd03a5e3dafad41c397063f9266c1b24fe5e53c14a08953f3154e"} Nov 02 14:13:11.600996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:11.600950 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" podStartSLOduration=1.600936573 podStartE2EDuration="1.600936573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:11.597426779 +0000 UTC m=+956.634531712" watchObservedRunningTime="2026-11-02 14:13:11.600936573 +0000 UTC m=+956.638041504" Nov 02 14:13:12.562289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:12.562258 4431 generic.go:356] "Generic (PLEG): container finished" podID="1f766458-1e64-452f-84f6-e2ff95fdd36a" containerID="d16bec4b9ef098e21063544b393670d9381b831b52cd83c6148d753da0f5ac26" exitCode=0 Nov 02 14:13:12.562648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:12.562346 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" event={"ID":"1f766458-1e64-452f-84f6-e2ff95fdd36a","Type":"ContainerDied","Data":"d16bec4b9ef098e21063544b393670d9381b831b52cd83c6148d753da0f5ac26"} Nov 02 14:13:13.717398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.717370 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:13.845800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.845717 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-xks-ca\") pod \"1f766458-1e64-452f-84f6-e2ff95fdd36a\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " Nov 02 14:13:13.845800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.845757 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f766458-1e64-452f-84f6-e2ff95fdd36a-kube-api-access-jszsm\" (UniqueName: \"kubernetes.io/projected/1f766458-1e64-452f-84f6-e2ff95fdd36a-kube-api-access-jszsm\") pod \"1f766458-1e64-452f-84f6-e2ff95fdd36a\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " Nov 02 14:13:13.845974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.845883 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-openshift-ca\") pod \"1f766458-1e64-452f-84f6-e2ff95fdd36a\" (UID: \"1f766458-1e64-452f-84f6-e2ff95fdd36a\") " Nov 02 14:13:13.846231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.846207 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-openshift-ca" pod "1f766458-1e64-452f-84f6-e2ff95fdd36a" (UID: "1f766458-1e64-452f-84f6-e2ff95fdd36a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:13.846433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.846412 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:13.847741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.847722 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f766458-1e64-452f-84f6-e2ff95fdd36a-kube-api-access-jszsm" pod "1f766458-1e64-452f-84f6-e2ff95fdd36a" (UID: "1f766458-1e64-452f-84f6-e2ff95fdd36a"). InnerVolumeSpecName "kube-api-access-jszsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:13.847844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.847750 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-xks-ca" pod "1f766458-1e64-452f-84f6-e2ff95fdd36a" (UID: "1f766458-1e64-452f-84f6-e2ff95fdd36a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:13.947589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.947560 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f766458-1e64-452f-84f6-e2ff95fdd36a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:13.947589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:13.947588 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jszsm\" (UniqueName: \"kubernetes.io/projected/1f766458-1e64-452f-84f6-e2ff95fdd36a-kube-api-access-jszsm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:14.388643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:14.388605 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:13:14.575996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:14.575961 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" Nov 02 14:13:14.575996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:14.575972 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz" event={"ID":"1f766458-1e64-452f-84f6-e2ff95fdd36a","Type":"ContainerDied","Data":"9f4bc1f6bd4cd03a5e3dafad41c397063f9266c1b24fe5e53c14a08953f3154e"} Nov 02 14:13:14.576182 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:14.576015 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f4bc1f6bd4cd03a5e3dafad41c397063f9266c1b24fe5e53c14a08953f3154e" Nov 02 14:13:19.078084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.078051 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq"] Nov 02 14:13:19.078655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.078532 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1f766458-1e64-452f-84f6-e2ff95fdd36a" containerName="revoke-keys" Nov 02 14:13:19.078655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.078546 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f766458-1e64-452f-84f6-e2ff95fdd36a" containerName="revoke-keys" Nov 02 14:13:19.078750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.078675 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f766458-1e64-452f-84f6-e2ff95fdd36a" containerName="revoke-keys" Nov 02 14:13:19.083086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.083060 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.105513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.105485 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq"] Nov 02 14:13:19.121906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.121880 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:13:19.122183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.122161 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:13:19.222627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.222591 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48589894-4950-4746-8581-a3f196d7193e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.222880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.222852 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48589894-4950-4746-8581-a3f196d7193e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.222937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.222906 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7zr7\" (UniqueName: \"kubernetes.io/projected/48589894-4950-4746-8581-a3f196d7193e-kube-api-access-c7zr7\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.324729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.324694 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48589894-4950-4746-8581-a3f196d7193e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.324729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.324732 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7zr7\" (UniqueName: \"kubernetes.io/projected/48589894-4950-4746-8581-a3f196d7193e-kube-api-access-c7zr7\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.324991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.324852 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48589894-4950-4746-8581-a3f196d7193e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.325406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.325378 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48589894-4950-4746-8581-a3f196d7193e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.327243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.327220 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48589894-4950-4746-8581-a3f196d7193e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.364439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.364386 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7zr7\" (UniqueName: \"kubernetes.io/projected/48589894-4950-4746-8581-a3f196d7193e-kube-api-access-c7zr7\") pod \"maas-api-revoke-sub-c01f49fb-492tq\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.398514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.398490 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:19.713617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:19.713578 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq"] Nov 02 14:13:19.716717 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:13:19.716682 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48589894_4950_4746_8581_a3f196d7193e.slice/crio-6f39c4a99766714bb1b41e749f9ddccb966da6f0c11d456030959964001986a0 WatchSource:0}: Error finding container 6f39c4a99766714bb1b41e749f9ddccb966da6f0c11d456030959964001986a0: Status 404 returned error can't find the container with id 6f39c4a99766714bb1b41e749f9ddccb966da6f0c11d456030959964001986a0 Nov 02 14:13:20.626523 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:20.626491 4431 generic.go:356] "Generic (PLEG): container finished" podID="48589894-4950-4746-8581-a3f196d7193e" containerID="8540db92fb944151846406e695f52737d983e492a8bef16cd266e4893fbd3878" exitCode=0 Nov 02 14:13:20.627018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:20.626573 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" event={"ID":"48589894-4950-4746-8581-a3f196d7193e","Type":"ContainerDied","Data":"8540db92fb944151846406e695f52737d983e492a8bef16cd266e4893fbd3878"} Nov 02 14:13:20.627018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:20.626628 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" event={"ID":"48589894-4950-4746-8581-a3f196d7193e","Type":"ContainerStarted","Data":"6f39c4a99766714bb1b41e749f9ddccb966da6f0c11d456030959964001986a0"} Nov 02 14:13:20.890067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:20.889908 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" podUID="dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" containerName="main" containerID="cri-o://165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf" gracePeriod=45 Nov 02 14:13:20.925894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:20.925862 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" podUID="dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": dial tcp 10.133.0.49:8000: connect: connection refused" Nov 02 14:13:21.159707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.159662 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:13:21.246472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.246445 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-model-cache\") pod \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " Nov 02 14:13:21.246617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.246490 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tls-certs\") pod \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " Nov 02 14:13:21.246617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.246544 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-dshm\") pod \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " Nov 02 14:13:21.246617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.246579 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-home\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-home\") pod \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " Nov 02 14:13:21.246829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.246637 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-kube-api-access-xsmwq\" (UniqueName: \"kubernetes.io/projected/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-kube-api-access-xsmwq\") pod \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " Nov 02 14:13:21.246829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.246681 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tmp-dir\") pod \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\" (UID: \"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9\") " Nov 02 14:13:21.246829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.246795 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-model-cache" pod "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" (UID: "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:13:21.246998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.246971 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-home" pod "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" (UID: "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:13:21.247203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.247181 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tmp-dir" pod "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" (UID: "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:13:21.247203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.247197 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:21.247354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.247217 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:21.248976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.248954 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tls-certs" pod "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" (UID: "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:21.248976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.248966 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-dshm" pod "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" (UID: "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:13:21.249286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.249273 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-kube-api-access-xsmwq" pod "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" (UID: "dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9"). InnerVolumeSpecName "kube-api-access-xsmwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:21.281424 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:13:21.281390 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:13:21.348201 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.348170 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsmwq\" (UniqueName: \"kubernetes.io/projected/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-kube-api-access-xsmwq\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:21.348201 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.348197 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:21.348340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.348213 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:21.348340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.348224 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:21.634822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.634732 4431 generic.go:356] "Generic (PLEG): container finished" podID="dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" containerID="165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf" exitCode=0 Nov 02 14:13:21.634822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.634803 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" Nov 02 14:13:21.634822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.634805 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" event={"ID":"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9","Type":"ContainerDied","Data":"165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf"} Nov 02 14:13:21.635342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.634856 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw" event={"ID":"dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9","Type":"ContainerDied","Data":"eba63562d72ec5ce6b48c06bc549e1b8d6b270896e2b760e3c3e99fe7ea5c3f4"} Nov 02 14:13:21.635342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.634872 4431 scope.go:122] "RemoveContainer" containerID="165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf" Nov 02 14:13:21.646177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.646157 4431 scope.go:122] "RemoveContainer" containerID="165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf" Nov 02 14:13:21.646439 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:13:21.646420 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf\": container with ID starting with 165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf not found: ID does not exist" containerID="165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf" Nov 02 14:13:21.646507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.646452 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf"} err="failed to get container status \"165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf\": rpc error: code = NotFound desc = could not find container \"165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf\": container with ID starting with 165586f08ff67b4bfce501ca72d15c04d39d8c2fe2bf060ec9d1833ba666fcdf not found: ID does not exist" Nov 02 14:13:21.696635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.696599 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw"] Nov 02 14:13:21.709383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.709360 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-758617/test-model-758617-kserve-5ff99b9f99-rk6tw"] Nov 02 14:13:21.788064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.788040 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:21.852958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.852925 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48589894-4950-4746-8581-a3f196d7193e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48589894-4950-4746-8581-a3f196d7193e-maas-api-xks-ca\") pod \"48589894-4950-4746-8581-a3f196d7193e\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " Nov 02 14:13:21.853106 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.853072 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48589894-4950-4746-8581-a3f196d7193e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48589894-4950-4746-8581-a3f196d7193e-maas-api-openshift-ca\") pod \"48589894-4950-4746-8581-a3f196d7193e\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " Nov 02 14:13:21.853182 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.853165 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48589894-4950-4746-8581-a3f196d7193e-kube-api-access-c7zr7\" (UniqueName: \"kubernetes.io/projected/48589894-4950-4746-8581-a3f196d7193e-kube-api-access-c7zr7\") pod \"48589894-4950-4746-8581-a3f196d7193e\" (UID: \"48589894-4950-4746-8581-a3f196d7193e\") " Nov 02 14:13:21.853609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.853471 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48589894-4950-4746-8581-a3f196d7193e-maas-api-openshift-ca" pod "48589894-4950-4746-8581-a3f196d7193e" (UID: "48589894-4950-4746-8581-a3f196d7193e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:13:21.853723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.853683 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48589894-4950-4746-8581-a3f196d7193e-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:21.855535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.855510 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48589894-4950-4746-8581-a3f196d7193e-maas-api-xks-ca" pod "48589894-4950-4746-8581-a3f196d7193e" (UID: "48589894-4950-4746-8581-a3f196d7193e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:13:21.855998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.855972 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48589894-4950-4746-8581-a3f196d7193e-kube-api-access-c7zr7" pod "48589894-4950-4746-8581-a3f196d7193e" (UID: "48589894-4950-4746-8581-a3f196d7193e"). InnerVolumeSpecName "kube-api-access-c7zr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:13:21.954397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.954369 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7zr7\" (UniqueName: \"kubernetes.io/projected/48589894-4950-4746-8581-a3f196d7193e-kube-api-access-c7zr7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:21.954542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:21.954400 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48589894-4950-4746-8581-a3f196d7193e-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:13:22.642818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:22.642765 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" Nov 02 14:13:22.643287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:22.642766 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq" event={"ID":"48589894-4950-4746-8581-a3f196d7193e","Type":"ContainerDied","Data":"6f39c4a99766714bb1b41e749f9ddccb966da6f0c11d456030959964001986a0"} Nov 02 14:13:22.643287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:22.642942 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f39c4a99766714bb1b41e749f9ddccb966da6f0c11d456030959964001986a0" Nov 02 14:13:23.077332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.077258 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" path="/var/lib/kubelet/pods/dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9/volumes" Nov 02 14:13:23.362183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.362071 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79"] Nov 02 14:13:23.362676 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.362656 4431 cpu_manager.go:414] "containerMap: removing container" podUID="dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" containerName="main" Nov 02 14:13:23.362832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.362677 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" containerName="main" Nov 02 14:13:23.362832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.362699 4431 cpu_manager.go:414] "containerMap: removing container" podUID="48589894-4950-4746-8581-a3f196d7193e" containerName="revoke-keys" Nov 02 14:13:23.362832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.362706 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48589894-4950-4746-8581-a3f196d7193e" containerName="revoke-keys" Nov 02 14:13:23.362995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.362875 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbda82a6-a948-4d2f-a49e-bc0ca92b4fb9" containerName="main" Nov 02 14:13:23.362995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.362889 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="48589894-4950-4746-8581-a3f196d7193e" containerName="revoke-keys" Nov 02 14:13:23.366851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.366823 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.381175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.381151 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d"] Nov 02 14:13:23.385020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.385002 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.396048 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.396024 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79"] Nov 02 14:13:23.408449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.408428 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d"] Nov 02 14:13:23.469256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469232 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.469370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469296 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469318 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469400 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtnb\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-kube-api-access-zvtnb\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.469443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469434 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.469518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469504 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469554 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469528 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469554 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469544 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpp7\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-kube-api-access-cwpp7\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469566 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469621 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.469701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469664 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.469701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469682 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.469765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469712 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469731 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.469878 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469859 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.469914 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469878 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469914 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469899 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59160f47-0e62-4d4a-97e3-dd608ffabf72-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.469977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.469920 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.571453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571431 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.571544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571469 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.571544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571496 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.571544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571522 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.571682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571566 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtnb\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-kube-api-access-zvtnb\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.571682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571609 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.571682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571656 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.571849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571682 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.571849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571748 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpp7\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-kube-api-access-cwpp7\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.571849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571818 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.571849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571840 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571872 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.572062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571889 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571908 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.572062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571917 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571943 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.571978 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572003 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.572062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572034 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572103 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572123 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.572480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572131 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.572480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572185 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59160f47-0e62-4d4a-97e3-dd608ffabf72-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.572480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572383 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-credential-socket\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572423 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-data\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.572806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572603 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572870 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572768 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-certs\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.572936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.572901 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59160f47-0e62-4d4a-97e3-dd608ffabf72-istiod-ca-cert\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.574306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.574280 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.574572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.574545 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.574681 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.574667 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-envoy\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.574955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.574936 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-podinfo\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.596713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.596691 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpp7\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-kube-api-access-cwpp7\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.596801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.596698 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.596850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.596809 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-token\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:23.597004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.596988 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtnb\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-kube-api-access-zvtnb\") pod \"e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.683995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.683967 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:23.701368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:23.701185 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:24.031235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.031161 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d"] Nov 02 14:13:24.044024 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:13:24.043995 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e3f5a6_db61_4edf_b6d7_e2839a55d659.slice/crio-b0a51c14538eac856abb7f3fb2f327162afe57a574512b702dec4f4035f93038 WatchSource:0}: Error finding container b0a51c14538eac856abb7f3fb2f327162afe57a574512b702dec4f4035f93038: Status 404 returned error can't find the container with id b0a51c14538eac856abb7f3fb2f327162afe57a574512b702dec4f4035f93038 Nov 02 14:13:24.044330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.044309 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79"] Nov 02 14:13:24.044822 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:13:24.044765 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59160f47_0e62_4d4a_97e3_dd608ffabf72.slice/crio-64aff65fa698312279e0eefb33b8c932537b9c4b1b3859fbf127dd7564985104 WatchSource:0}: Error finding container 64aff65fa698312279e0eefb33b8c932537b9c4b1b3859fbf127dd7564985104: Status 404 returned error can't find the container with id 64aff65fa698312279e0eefb33b8c932537b9c4b1b3859fbf127dd7564985104 Nov 02 14:13:24.046678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.046642 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:24.046739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.046723 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:24.046791 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.046759 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:24.047339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.047232 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:24.047339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.047309 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:24.047449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.047352 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:24.659288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.659256 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" event={"ID":"59160f47-0e62-4d4a-97e3-dd608ffabf72","Type":"ContainerStarted","Data":"bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024"} Nov 02 14:13:24.659458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.659311 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" event={"ID":"59160f47-0e62-4d4a-97e3-dd608ffabf72","Type":"ContainerStarted","Data":"64aff65fa698312279e0eefb33b8c932537b9c4b1b3859fbf127dd7564985104"} Nov 02 14:13:24.660971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.660945 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" event={"ID":"f7e3f5a6-db61-4edf-b6d7-e2839a55d659","Type":"ContainerStarted","Data":"18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1"} Nov 02 14:13:24.661088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.660987 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" event={"ID":"f7e3f5a6-db61-4edf-b6d7-e2839a55d659","Type":"ContainerStarted","Data":"b0a51c14538eac856abb7f3fb2f327162afe57a574512b702dec4f4035f93038"} Nov 02 14:13:24.685045 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.685021 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:24.701763 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.701727 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:24.721430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.721386 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podStartSLOduration=1.721373883 podStartE2EDuration="1.721373883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:24.719896726 +0000 UTC m=+969.757001658" watchObservedRunningTime="2026-11-02 14:13:24.721373883 +0000 UTC m=+969.758478814" Nov 02 14:13:24.773945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:24.773878 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podStartSLOduration=1.773859006 podStartE2EDuration="1.773859006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:24.768058217 +0000 UTC m=+969.805163150" watchObservedRunningTime="2026-11-02 14:13:24.773859006 +0000 UTC m=+969.810963937" Nov 02 14:13:25.690808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:25.690672 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:13:25.690808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:25.690756 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:13:25.719805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:25.709715 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 14:13:25.719805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:25.709788 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 14:13:26.685803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:26.684635 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:13:26.685803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:26.684692 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:13:26.710818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:26.707185 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 14:13:26.710818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:26.707239 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 14:13:27.693301 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:27.693262 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:13:27.693605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:27.693575 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:13:27.706337 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:27.706305 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 14:13:27.706562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:27.706536 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 14:13:28.421968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:28.421926 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:28.553836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:28.553505 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:28.680224 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:28.679862 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:28.708962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:28.708523 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:13:28.708962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:28.708581 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:13:28.709295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:28.709270 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 14:13:28.709440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:28.709415 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 14:13:29.062412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:29.062309 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:29.422940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:29.422893 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 14:13:29.424278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:29.424257 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:13:29.689855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:29.684332 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:13:29.689855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:29.684388 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:13:29.706799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:29.703913 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:29.761801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:29.760162 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 14:13:29.761801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:29.760220 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 14:13:30.090690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.073690 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:30.103863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.100961 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:13:30.447122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.447079 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:30.568189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.568133 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 14:13:30.729456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.729358 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:13:30.747542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.747501 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:13:30.752806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.749976 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:13:30.752806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.750060 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:13:30.809004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.808975 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 14:13:30.809311 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.809268 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 14:13:31.006853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:30.975277 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:13:31.064804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:31.038916 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:13:31.222724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:31.222692 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:13:31.687191 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:31.687152 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 14:13:31.687824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:31.687758 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 14:13:31.703105 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:31.703079 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 14:13:31.703286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:31.703260 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 14:13:32.756497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:32.756471 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:32.757669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:32.757649 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:32.758097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:32.758080 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:13:33.644304 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:33.644268 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:33.644986 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:33.644963 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:33.645839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:33.645820 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:13:34.854218 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:34.854187 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:13:38.705213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.705181 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc"] Nov 02 14:13:38.710695 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.710669 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.732575 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.732551 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc"] Nov 02 14:13:38.740141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.740112 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4be645\"/\"distinct-w2-4be645-kserve-self-signed-certs\"" Nov 02 14:13:38.850614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.850577 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6zr\" (UniqueName: \"kubernetes.io/projected/e9e2ef36-f412-4e56-a735-8a035e9fddbc-kube-api-access-kn6zr\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.850795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.850623 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-model-cache\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.850881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.850787 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-home\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.850881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.850841 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-dshm\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.850985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.850879 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tmp-dir\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.850985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.850935 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tls-certs\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.954344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.954303 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-home\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.954344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.954352 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-dshm\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.954591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.954375 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tmp-dir\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.954591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.954408 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tls-certs\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.954591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.954536 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6zr\" (UniqueName: \"kubernetes.io/projected/e9e2ef36-f412-4e56-a735-8a035e9fddbc-kube-api-access-kn6zr\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.954591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.954563 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-model-cache\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.955346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.955280 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tmp-dir\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.955544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.955514 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-home\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.955695 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.955628 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-model-cache\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.960090 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.959618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tls-certs\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.960936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.960915 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-dshm\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:38.982901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:38.982865 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6zr\" (UniqueName: \"kubernetes.io/projected/e9e2ef36-f412-4e56-a735-8a035e9fddbc-kube-api-access-kn6zr\") pod \"distinct-w2-4be645-kserve-85c98cc649-rq7nc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:39.024200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:39.024170 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:39.352759 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:13:39.352724 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9e2ef36_f412_4e56_a735_8a035e9fddbc.slice/crio-7d228d89a6c74e945993639d0e319db24e92dc630d5123e71b7f0fd588b0e814 WatchSource:0}: Error finding container 7d228d89a6c74e945993639d0e319db24e92dc630d5123e71b7f0fd588b0e814: Status 404 returned error can't find the container with id 7d228d89a6c74e945993639d0e319db24e92dc630d5123e71b7f0fd588b0e814 Nov 02 14:13:39.354034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:39.354012 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc"] Nov 02 14:13:40.123401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.123372 4431 generic.go:356] "Generic (PLEG): container finished" podID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerID="3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b" exitCode=0 Nov 02 14:13:40.123847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.123446 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" event={"ID":"273ac84f-f3ab-44e7-998b-4fa269667ca0","Type":"ContainerDied","Data":"3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b"} Nov 02 14:13:40.123922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.123868 4431 scope.go:122] "RemoveContainer" containerID="3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b" Nov 02 14:13:40.125245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.125218 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:40.125245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.125239 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" event={"ID":"e9e2ef36-f412-4e56-a735-8a035e9fddbc","Type":"ContainerStarted","Data":"e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636"} Nov 02 14:13:40.125401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.125265 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" event={"ID":"e9e2ef36-f412-4e56-a735-8a035e9fddbc","Type":"ContainerStarted","Data":"7d228d89a6c74e945993639d0e319db24e92dc630d5123e71b7f0fd588b0e814"} Nov 02 14:13:40.125401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.125283 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:40.125401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.125316 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:13:40.236581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:40.236484 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podStartSLOduration=2.2364651 podStartE2EDuration="2.2364651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:40.23249439 +0000 UTC m=+985.269599324" watchObservedRunningTime="2026-11-02 14:13:40.2364651 +0000 UTC m=+985.273570034" Nov 02 14:13:41.024615 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:41.024567 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:41.138507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:41.138479 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" event={"ID":"273ac84f-f3ab-44e7-998b-4fa269667ca0","Type":"ContainerStarted","Data":"46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0"} Nov 02 14:13:41.927689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:41.927658 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:13:41.928873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:41.928845 4431 patch_prober.go:29] interesting pod/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:13:41.929006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:41.928892 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:13:42.927932 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:42.927897 4431 patch_prober.go:29] interesting pod/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:13:42.928323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:42.927958 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:13:43.929190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:43.927610 4431 patch_prober.go:29] interesting pod/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:13:43.929190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:43.927663 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:13:44.461492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:44.461458 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:13:44.928063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:44.928032 4431 patch_prober.go:29] interesting pod/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:13:44.928232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:44.928085 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:13:45.928307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:45.928269 4431 patch_prober.go:29] interesting pod/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 14:13:45.928956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:45.928908 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 14:13:46.947392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:46.947365 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:13:46.951986 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:46.951957 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:13:49.024508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:49.024474 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:49.037353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:49.037332 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:49.229721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:49.229694 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:13:51.284650 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:13:51.284610 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:13:56.712641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:13:56.712597 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:14:08.593486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.593402 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb"] Nov 02 14:14:08.598316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.598289 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.620352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.620321 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb"] Nov 02 14:14:08.626667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.626625 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:14:08.626667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.626648 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:14:08.635306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.635281 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.635429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.635337 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.635490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.635447 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vqjd\" (UniqueName: \"kubernetes.io/projected/8a7ce241-aa59-4811-81e6-d61daa007fc0-kube-api-access-6vqjd\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.736415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.736382 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.736650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.736439 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.736650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.736478 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vqjd\" (UniqueName: \"kubernetes.io/projected/8a7ce241-aa59-4811-81e6-d61daa007fc0-kube-api-access-6vqjd\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.737110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.737082 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.738959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.738937 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.760947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.760922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vqjd\" (UniqueName: \"kubernetes.io/projected/8a7ce241-aa59-4811-81e6-d61daa007fc0-kube-api-access-6vqjd\") pod \"maas-api-revoke-sub-d0e2dffb-sq5fb\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:08.917056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:08.917029 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:09.302199 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:09.302169 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb"] Nov 02 14:14:09.302358 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:14:09.302179 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7ce241_aa59_4811_81e6_d61daa007fc0.slice/crio-3b8d59455598b64d501c757d16b7d9e867de3e0f8be8ff79d01449a0d0f5be43 WatchSource:0}: Error finding container 3b8d59455598b64d501c757d16b7d9e867de3e0f8be8ff79d01449a0d0f5be43: Status 404 returned error can't find the container with id 3b8d59455598b64d501c757d16b7d9e867de3e0f8be8ff79d01449a0d0f5be43 Nov 02 14:14:09.369649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:09.369621 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" event={"ID":"8a7ce241-aa59-4811-81e6-d61daa007fc0","Type":"ContainerStarted","Data":"3b8d59455598b64d501c757d16b7d9e867de3e0f8be8ff79d01449a0d0f5be43"} Nov 02 14:14:10.429806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:10.429747 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" event={"ID":"8a7ce241-aa59-4811-81e6-d61daa007fc0","Type":"ContainerStarted","Data":"44d1b1bb581fb9c4b00feade42ce1d470ff2b2f3f4ac2b4baee87b7f6a0e7954"} Nov 02 14:14:10.517739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:10.517671 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" podStartSLOduration=2.517649878 podStartE2EDuration="2.517649878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:10.51610016 +0000 UTC m=+1015.553205094" watchObservedRunningTime="2026-11-02 14:14:10.517649878 +0000 UTC m=+1015.554754811" Nov 02 14:14:11.439332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:11.439291 4431 generic.go:356] "Generic (PLEG): container finished" podID="8a7ce241-aa59-4811-81e6-d61daa007fc0" containerID="44d1b1bb581fb9c4b00feade42ce1d470ff2b2f3f4ac2b4baee87b7f6a0e7954" exitCode=0 Nov 02 14:14:11.439837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:11.439473 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" event={"ID":"8a7ce241-aa59-4811-81e6-d61daa007fc0","Type":"ContainerDied","Data":"44d1b1bb581fb9c4b00feade42ce1d470ff2b2f3f4ac2b4baee87b7f6a0e7954"} Nov 02 14:14:12.601546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.601519 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:12.684608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.684575 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7ce241-aa59-4811-81e6-d61daa007fc0-kube-api-access-6vqjd\" (UniqueName: \"kubernetes.io/projected/8a7ce241-aa59-4811-81e6-d61daa007fc0-kube-api-access-6vqjd\") pod \"8a7ce241-aa59-4811-81e6-d61daa007fc0\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " Nov 02 14:14:12.684755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.684646 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-xks-ca\") pod \"8a7ce241-aa59-4811-81e6-d61daa007fc0\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " Nov 02 14:14:12.684885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.684861 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-openshift-ca\") pod \"8a7ce241-aa59-4811-81e6-d61daa007fc0\" (UID: \"8a7ce241-aa59-4811-81e6-d61daa007fc0\") " Nov 02 14:14:12.685260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.685235 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-openshift-ca" pod "8a7ce241-aa59-4811-81e6-d61daa007fc0" (UID: "8a7ce241-aa59-4811-81e6-d61daa007fc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:12.686902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.686873 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7ce241-aa59-4811-81e6-d61daa007fc0-kube-api-access-6vqjd" pod "8a7ce241-aa59-4811-81e6-d61daa007fc0" (UID: "8a7ce241-aa59-4811-81e6-d61daa007fc0"). InnerVolumeSpecName "kube-api-access-6vqjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:12.686998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.686948 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-xks-ca" pod "8a7ce241-aa59-4811-81e6-d61daa007fc0" (UID: "8a7ce241-aa59-4811-81e6-d61daa007fc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:14:12.786381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.786313 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vqjd\" (UniqueName: \"kubernetes.io/projected/8a7ce241-aa59-4811-81e6-d61daa007fc0-kube-api-access-6vqjd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:12.786381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.786341 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:12.786381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:12.786354 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7ce241-aa59-4811-81e6-d61daa007fc0-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:13.457374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:13.457340 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" event={"ID":"8a7ce241-aa59-4811-81e6-d61daa007fc0","Type":"ContainerDied","Data":"3b8d59455598b64d501c757d16b7d9e867de3e0f8be8ff79d01449a0d0f5be43"} Nov 02 14:14:13.457531 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:13.457397 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8d59455598b64d501c757d16b7d9e867de3e0f8be8ff79d01449a0d0f5be43" Nov 02 14:14:13.457531 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:13.457349 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb" Nov 02 14:14:16.043356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:16.043326 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:14:21.129711 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.129672 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm"] Nov 02 14:14:21.130278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.130260 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8a7ce241-aa59-4811-81e6-d61daa007fc0" containerName="revoke-keys" Nov 02 14:14:21.130278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.130277 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7ce241-aa59-4811-81e6-d61daa007fc0" containerName="revoke-keys" Nov 02 14:14:21.130416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.130406 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7ce241-aa59-4811-81e6-d61daa007fc0" containerName="revoke-keys" Nov 02 14:14:21.133489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.133468 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.150075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.149983 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:14:21.156121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.156098 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm"] Nov 02 14:14:21.162730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.162707 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:14:21.215666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.215634 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cpw5\" (UniqueName: \"kubernetes.io/projected/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-kube-api-access-9cpw5\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.215832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.215696 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.215832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.215771 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.283840 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:14:21.283795 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:14:21.317676 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.317646 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cpw5\" (UniqueName: \"kubernetes.io/projected/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-kube-api-access-9cpw5\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.317845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.317706 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.317915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.317844 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.318385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.318365 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.320139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.320120 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.342596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.342569 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cpw5\" (UniqueName: \"kubernetes.io/projected/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-kube-api-access-9cpw5\") pod \"maas-api-revoke-sub-e5162db5-4qgsm\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.444924 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.444894 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:21.765015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:21.764985 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm"] Nov 02 14:14:21.768101 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:14:21.768072 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc3fe90_0460_4eff_83ef_1fe08c2f2f3b.slice/crio-b716f981dccb3acf9681841a62ab9ff7f66717fe2c3ad65ecec105f8870a7108 WatchSource:0}: Error finding container b716f981dccb3acf9681841a62ab9ff7f66717fe2c3ad65ecec105f8870a7108: Status 404 returned error can't find the container with id b716f981dccb3acf9681841a62ab9ff7f66717fe2c3ad65ecec105f8870a7108 Nov 02 14:14:22.558874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:22.558832 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" event={"ID":"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b","Type":"ContainerStarted","Data":"912bc5486debd14728fef4c588c675cc44170629b9889b4a5432c1ac2b6e0101"} Nov 02 14:14:22.559245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:22.558892 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" event={"ID":"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b","Type":"ContainerStarted","Data":"b716f981dccb3acf9681841a62ab9ff7f66717fe2c3ad65ecec105f8870a7108"} Nov 02 14:14:22.605440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:22.605390 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" podStartSLOduration=1.605377531 podStartE2EDuration="1.605377531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:22.601198427 +0000 UTC m=+1027.638303364" watchObservedRunningTime="2026-11-02 14:14:22.605377531 +0000 UTC m=+1027.642482462" Nov 02 14:14:23.566768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:23.566735 4431 generic.go:356] "Generic (PLEG): container finished" podID="8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" containerID="912bc5486debd14728fef4c588c675cc44170629b9889b4a5432c1ac2b6e0101" exitCode=0 Nov 02 14:14:23.567160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:23.566806 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" event={"ID":"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b","Type":"ContainerDied","Data":"912bc5486debd14728fef4c588c675cc44170629b9889b4a5432c1ac2b6e0101"} Nov 02 14:14:24.783693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.783669 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:24.858297 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.858231 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-openshift-ca\") pod \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " Nov 02 14:14:24.858411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.858315 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-xks-ca\") pod \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " Nov 02 14:14:24.858411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.858392 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-kube-api-access-9cpw5\" (UniqueName: \"kubernetes.io/projected/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-kube-api-access-9cpw5\") pod \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\" (UID: \"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b\") " Nov 02 14:14:24.858669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.858643 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-openshift-ca" pod "8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" (UID: "8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:24.858862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.858842 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:24.860400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.860375 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-xks-ca" pod "8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" (UID: "8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:14:24.860536 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.860514 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-kube-api-access-9cpw5" pod "8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" (UID: "8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b"). InnerVolumeSpecName "kube-api-access-9cpw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:24.959634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.959612 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:24.959634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:24.959634 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cpw5\" (UniqueName: \"kubernetes.io/projected/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b-kube-api-access-9cpw5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:25.594306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:25.594269 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:14:25.594306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:25.594294 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" event={"ID":"8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b","Type":"ContainerDied","Data":"b716f981dccb3acf9681841a62ab9ff7f66717fe2c3ad65ecec105f8870a7108"} Nov 02 14:14:25.594514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:25.594352 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b716f981dccb3acf9681841a62ab9ff7f66717fe2c3ad65ecec105f8870a7108" Nov 02 14:14:29.569856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.569125 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757"] Nov 02 14:14:29.570448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.570019 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" containerName="revoke-keys" Nov 02 14:14:29.570448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.570041 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" containerName="revoke-keys" Nov 02 14:14:29.570448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.570162 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" containerName="revoke-keys" Nov 02 14:14:29.602802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.584026 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.646800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.633985 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:14:29.646800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.634073 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:14:29.650799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.647519 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757"] Nov 02 14:14:29.687799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.686913 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.687799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.686963 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.687799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.687139 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlgc\" (UniqueName: \"kubernetes.io/projected/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-kube-api-access-lzlgc\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.813802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.809432 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlgc\" (UniqueName: \"kubernetes.io/projected/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-kube-api-access-lzlgc\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.813802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.809525 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.813802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.809562 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.843175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.843106 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.857798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.851391 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.913799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.913246 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlgc\" (UniqueName: \"kubernetes.io/projected/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-kube-api-access-lzlgc\") pod \"maas-api-revoke-sub-4171a261-kx757\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:29.983756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:29.983725 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:30.028972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:30.026889 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:30.168362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:30.168314 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 14:14:30.237200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:30.237157 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:14:30.453967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:30.453863 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 14:14:30.556969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:30.547405 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 14:14:31.036318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:31.036285 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757"] Nov 02 14:14:31.930582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:31.930543 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" event={"ID":"626d7ce8-52ee-4ed3-9166-c2c4b4f82927","Type":"ContainerStarted","Data":"4f98bc4ec99c30a9362442030d7f3abd45a512d6a4c6a8f8f772570dc09ddd46"} Nov 02 14:14:31.930757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:31.930611 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" event={"ID":"626d7ce8-52ee-4ed3-9166-c2c4b4f82927","Type":"ContainerStarted","Data":"95a6c9524ea7098eea22781fa1c39f8c9bf5f4727d23cdba08bcf60f0126baed"} Nov 02 14:14:31.978166 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:31.978110 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" podStartSLOduration=2.978091828 podStartE2EDuration="2.978091828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:31.972854349 +0000 UTC m=+1037.009959282" watchObservedRunningTime="2026-11-02 14:14:31.978091828 +0000 UTC m=+1037.015196767" Nov 02 14:14:32.940438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:32.940406 4431 generic.go:356] "Generic (PLEG): container finished" podID="626d7ce8-52ee-4ed3-9166-c2c4b4f82927" containerID="4f98bc4ec99c30a9362442030d7f3abd45a512d6a4c6a8f8f772570dc09ddd46" exitCode=0 Nov 02 14:14:32.941053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:32.940742 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" event={"ID":"626d7ce8-52ee-4ed3-9166-c2c4b4f82927","Type":"ContainerDied","Data":"4f98bc4ec99c30a9362442030d7f3abd45a512d6a4c6a8f8f772570dc09ddd46"} Nov 02 14:14:34.095165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.095137 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:34.140130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.140107 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-xks-ca\") pod \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " Nov 02 14:14:34.140267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.140238 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-kube-api-access-lzlgc\" (UniqueName: \"kubernetes.io/projected/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-kube-api-access-lzlgc\") pod \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " Nov 02 14:14:34.140331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.140291 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-openshift-ca\") pod \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\" (UID: \"626d7ce8-52ee-4ed3-9166-c2c4b4f82927\") " Nov 02 14:14:34.140635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.140607 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-openshift-ca" pod "626d7ce8-52ee-4ed3-9166-c2c4b4f82927" (UID: "626d7ce8-52ee-4ed3-9166-c2c4b4f82927"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:34.142251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.142220 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-kube-api-access-lzlgc" pod "626d7ce8-52ee-4ed3-9166-c2c4b4f82927" (UID: "626d7ce8-52ee-4ed3-9166-c2c4b4f82927"). InnerVolumeSpecName "kube-api-access-lzlgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:34.142335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.142226 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-xks-ca" pod "626d7ce8-52ee-4ed3-9166-c2c4b4f82927" (UID: "626d7ce8-52ee-4ed3-9166-c2c4b4f82927"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:14:34.241320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.241258 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzlgc\" (UniqueName: \"kubernetes.io/projected/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-kube-api-access-lzlgc\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:34.241320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.241284 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:34.241320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.241294 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/626d7ce8-52ee-4ed3-9166-c2c4b4f82927-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:34.955642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.955611 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" Nov 02 14:14:34.955858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.955654 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757" event={"ID":"626d7ce8-52ee-4ed3-9166-c2c4b4f82927","Type":"ContainerDied","Data":"95a6c9524ea7098eea22781fa1c39f8c9bf5f4727d23cdba08bcf60f0126baed"} Nov 02 14:14:34.955858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:34.955733 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a6c9524ea7098eea22781fa1c39f8c9bf5f4727d23cdba08bcf60f0126baed" Nov 02 14:14:41.380314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.380283 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd"] Nov 02 14:14:41.380839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.380820 4431 cpu_manager.go:414] "containerMap: removing container" podUID="626d7ce8-52ee-4ed3-9166-c2c4b4f82927" containerName="revoke-keys" Nov 02 14:14:41.380888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.380840 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="626d7ce8-52ee-4ed3-9166-c2c4b4f82927" containerName="revoke-keys" Nov 02 14:14:41.380986 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.380977 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="626d7ce8-52ee-4ed3-9166-c2c4b4f82927" containerName="revoke-keys" Nov 02 14:14:41.384548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.384532 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.405608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.405583 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd"] Nov 02 14:14:41.413449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.413431 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:14:41.413595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.413455 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:14:41.419582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.419540 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crrnw\" (UniqueName: \"kubernetes.io/projected/09243e92-8800-4b16-a99d-30da706f7fd4-kube-api-access-crrnw\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.419814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.419759 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.419924 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.419865 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.519542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.519517 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:14:41.520557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.520538 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.520738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.520717 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crrnw\" (UniqueName: \"kubernetes.io/projected/09243e92-8800-4b16-a99d-30da706f7fd4-kube-api-access-crrnw\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.520819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.520764 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.521385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.521366 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.523061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.523038 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.545439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.545417 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crrnw\" (UniqueName: \"kubernetes.io/projected/09243e92-8800-4b16-a99d-30da706f7fd4-kube-api-access-crrnw\") pod \"maas-api-revoke-sub-d196c714-fw6cd\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:41.697401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:41.697375 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:42.004633 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:14:42.004598 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09243e92_8800_4b16_a99d_30da706f7fd4.slice/crio-f54fd2d0a42be0b94eaae331e64f3ca5e728fb879251a02811c1abdcd984824d WatchSource:0}: Error finding container f54fd2d0a42be0b94eaae331e64f3ca5e728fb879251a02811c1abdcd984824d: Status 404 returned error can't find the container with id f54fd2d0a42be0b94eaae331e64f3ca5e728fb879251a02811c1abdcd984824d Nov 02 14:14:42.004633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:42.004609 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd"] Nov 02 14:14:43.013047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:43.013012 4431 generic.go:356] "Generic (PLEG): container finished" podID="09243e92-8800-4b16-a99d-30da706f7fd4" containerID="146de96eb84824d2cfd463f2102abeeacfa8e72d5969b06112dbf9285536f7a1" exitCode=0 Nov 02 14:14:43.013401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:43.013094 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" event={"ID":"09243e92-8800-4b16-a99d-30da706f7fd4","Type":"ContainerDied","Data":"146de96eb84824d2cfd463f2102abeeacfa8e72d5969b06112dbf9285536f7a1"} Nov 02 14:14:43.013401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:43.013144 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" event={"ID":"09243e92-8800-4b16-a99d-30da706f7fd4","Type":"ContainerStarted","Data":"f54fd2d0a42be0b94eaae331e64f3ca5e728fb879251a02811c1abdcd984824d"} Nov 02 14:14:44.187945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.187920 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:44.254289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.254256 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-xks-ca\") pod \"09243e92-8800-4b16-a99d-30da706f7fd4\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " Nov 02 14:14:44.254472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.254326 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-openshift-ca\") pod \"09243e92-8800-4b16-a99d-30da706f7fd4\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " Nov 02 14:14:44.254549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.254473 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09243e92-8800-4b16-a99d-30da706f7fd4-kube-api-access-crrnw\" (UniqueName: \"kubernetes.io/projected/09243e92-8800-4b16-a99d-30da706f7fd4-kube-api-access-crrnw\") pod \"09243e92-8800-4b16-a99d-30da706f7fd4\" (UID: \"09243e92-8800-4b16-a99d-30da706f7fd4\") " Nov 02 14:14:44.254691 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.254654 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-openshift-ca" pod "09243e92-8800-4b16-a99d-30da706f7fd4" (UID: "09243e92-8800-4b16-a99d-30da706f7fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:44.255031 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.255006 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:44.256416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.256392 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-xks-ca" pod "09243e92-8800-4b16-a99d-30da706f7fd4" (UID: "09243e92-8800-4b16-a99d-30da706f7fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:14:44.256534 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.256518 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09243e92-8800-4b16-a99d-30da706f7fd4-kube-api-access-crrnw" pod "09243e92-8800-4b16-a99d-30da706f7fd4" (UID: "09243e92-8800-4b16-a99d-30da706f7fd4"). InnerVolumeSpecName "kube-api-access-crrnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:44.356066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.356009 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crrnw\" (UniqueName: \"kubernetes.io/projected/09243e92-8800-4b16-a99d-30da706f7fd4-kube-api-access-crrnw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:44.356066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.356033 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09243e92-8800-4b16-a99d-30da706f7fd4-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:44.499805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.499767 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf"] Nov 02 14:14:44.500366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.500352 4431 cpu_manager.go:414] "containerMap: removing container" podUID="09243e92-8800-4b16-a99d-30da706f7fd4" containerName="revoke-keys" Nov 02 14:14:44.500407 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.500367 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09243e92-8800-4b16-a99d-30da706f7fd4" containerName="revoke-keys" Nov 02 14:14:44.500486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.500476 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="09243e92-8800-4b16-a99d-30da706f7fd4" containerName="revoke-keys" Nov 02 14:14:44.504578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.504551 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.529430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.529396 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf"] Nov 02 14:14:44.558315 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.558284 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g8c\" (UniqueName: \"kubernetes.io/projected/6029d56a-5483-4a75-be08-bb6be38b21ac-kube-api-access-24g8c\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.558486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.558409 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.558556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.558486 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.660067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.660042 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g8c\" (UniqueName: \"kubernetes.io/projected/6029d56a-5483-4a75-be08-bb6be38b21ac-kube-api-access-24g8c\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.660209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.660116 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.660209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.660155 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.660921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.660896 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.662299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.662281 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.685739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.685710 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g8c\" (UniqueName: \"kubernetes.io/projected/6029d56a-5483-4a75-be08-bb6be38b21ac-kube-api-access-24g8c\") pod \"maas-api-revoke-sub-9641e5ba-rlhgf\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:44.826490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:44.826462 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:45.028033 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:45.027958 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" Nov 02 14:14:45.028033 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:45.027993 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd" event={"ID":"09243e92-8800-4b16-a99d-30da706f7fd4","Type":"ContainerDied","Data":"f54fd2d0a42be0b94eaae331e64f3ca5e728fb879251a02811c1abdcd984824d"} Nov 02 14:14:45.028223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:45.028048 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f54fd2d0a42be0b94eaae331e64f3ca5e728fb879251a02811c1abdcd984824d" Nov 02 14:14:45.129745 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:14:45.129718 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6029d56a_5483_4a75_be08_bb6be38b21ac.slice/crio-435eab6bb14ac917dbc15cc30dfcf7c5c88f7c0f049ed8f70b26933860143ac8 WatchSource:0}: Error finding container 435eab6bb14ac917dbc15cc30dfcf7c5c88f7c0f049ed8f70b26933860143ac8: Status 404 returned error can't find the container with id 435eab6bb14ac917dbc15cc30dfcf7c5c88f7c0f049ed8f70b26933860143ac8 Nov 02 14:14:45.130383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:45.130361 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf"] Nov 02 14:14:46.037819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:46.037772 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" event={"ID":"6029d56a-5483-4a75-be08-bb6be38b21ac","Type":"ContainerStarted","Data":"2f7461d3b8aea716628bd434bfb640d5e14dc27d0767bfc10a251cceeb434c4b"} Nov 02 14:14:46.038285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:46.037851 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" event={"ID":"6029d56a-5483-4a75-be08-bb6be38b21ac","Type":"ContainerStarted","Data":"435eab6bb14ac917dbc15cc30dfcf7c5c88f7c0f049ed8f70b26933860143ac8"} Nov 02 14:14:46.088821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:46.086902 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" podStartSLOduration=2.086887258 podStartE2EDuration="2.086887258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:46.081084744 +0000 UTC m=+1051.118189681" watchObservedRunningTime="2026-11-02 14:14:46.086887258 +0000 UTC m=+1051.123992190" Nov 02 14:14:47.045252 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.045216 4431 generic.go:356] "Generic (PLEG): container finished" podID="6029d56a-5483-4a75-be08-bb6be38b21ac" containerID="2f7461d3b8aea716628bd434bfb640d5e14dc27d0767bfc10a251cceeb434c4b" exitCode=0 Nov 02 14:14:47.045649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.045264 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" event={"ID":"6029d56a-5483-4a75-be08-bb6be38b21ac","Type":"ContainerDied","Data":"2f7461d3b8aea716628bd434bfb640d5e14dc27d0767bfc10a251cceeb434c4b"} Nov 02 14:14:47.229255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.229155 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz"] Nov 02 14:14:47.239648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.238545 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.253629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.253597 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz"] Nov 02 14:14:47.264898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.264875 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-de8cc7\"/\"praxis-route-de8cc7-kserve-self-signed-certs\"" Nov 02 14:14:47.264898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.264875 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-de8cc7\"/\"default-dockercfg-ljt8z\"" Nov 02 14:14:47.277468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.277448 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-de8cc7\"/\"kube-root-ca.crt\"" Nov 02 14:14:47.277597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.277581 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-de8cc7\"/\"openshift-service-ca.crt\"" Nov 02 14:14:47.296531 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.296463 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-tmp-dir\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.296645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.296549 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-home\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.296645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.296576 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5540857d-32be-4620-8c54-872b9a767f3e-tls-certs\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.296645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.296607 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-model-cache\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.296820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.296748 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-dshm\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.296877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.296852 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtwb\" (UniqueName: \"kubernetes.io/projected/5540857d-32be-4620-8c54-872b9a767f3e-kube-api-access-xwtwb\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.397926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.397883 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-home\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.397926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.397924 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5540857d-32be-4620-8c54-872b9a767f3e-tls-certs\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.398162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.397952 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-model-cache\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.398162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.398015 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-dshm\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.398162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.398052 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtwb\" (UniqueName: \"kubernetes.io/projected/5540857d-32be-4620-8c54-872b9a767f3e-kube-api-access-xwtwb\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.398317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.398221 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-tmp-dir\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.398369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.398347 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-home\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.398429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.398396 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-model-cache\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.398637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.398618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-tmp-dir\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.400504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.400474 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-dshm\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.400644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.400521 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5540857d-32be-4620-8c54-872b9a767f3e-tls-certs\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.430443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.430415 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtwb\" (UniqueName: \"kubernetes.io/projected/5540857d-32be-4620-8c54-872b9a767f3e-kube-api-access-xwtwb\") pod \"praxis-route-de8cc7-kserve-84d7894c89-xmwtz\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.555598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.555515 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:47.962112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:47.962086 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz"] Nov 02 14:14:47.962653 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:14:47.962632 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5540857d_32be_4620_8c54_872b9a767f3e.slice/crio-2cd21e069eb582c389175ef8d9fcabec5b2809ed863c0c50b441267f99da12cf WatchSource:0}: Error finding container 2cd21e069eb582c389175ef8d9fcabec5b2809ed863c0c50b441267f99da12cf: Status 404 returned error can't find the container with id 2cd21e069eb582c389175ef8d9fcabec5b2809ed863c0c50b441267f99da12cf Nov 02 14:14:48.057377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.057346 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" event={"ID":"5540857d-32be-4620-8c54-872b9a767f3e","Type":"ContainerStarted","Data":"4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee"} Nov 02 14:14:48.057860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.057385 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" event={"ID":"5540857d-32be-4620-8c54-872b9a767f3e","Type":"ContainerStarted","Data":"2cd21e069eb582c389175ef8d9fcabec5b2809ed863c0c50b441267f99da12cf"} Nov 02 14:14:48.107075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.106403 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" podStartSLOduration=1.106387504 podStartE2EDuration="1.106387504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:48.104463603 +0000 UTC m=+1053.141568563" watchObservedRunningTime="2026-11-02 14:14:48.106387504 +0000 UTC m=+1053.143492441" Nov 02 14:14:48.214348 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.214289 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:48.332387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.332356 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-xks-ca\") pod \"6029d56a-5483-4a75-be08-bb6be38b21ac\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " Nov 02 14:14:48.332594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.332403 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6029d56a-5483-4a75-be08-bb6be38b21ac-kube-api-access-24g8c\" (UniqueName: \"kubernetes.io/projected/6029d56a-5483-4a75-be08-bb6be38b21ac-kube-api-access-24g8c\") pod \"6029d56a-5483-4a75-be08-bb6be38b21ac\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " Nov 02 14:14:48.332655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.332643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-openshift-ca\") pod \"6029d56a-5483-4a75-be08-bb6be38b21ac\" (UID: \"6029d56a-5483-4a75-be08-bb6be38b21ac\") " Nov 02 14:14:48.333154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.333119 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-openshift-ca" pod "6029d56a-5483-4a75-be08-bb6be38b21ac" (UID: "6029d56a-5483-4a75-be08-bb6be38b21ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:48.334959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.334930 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-xks-ca" pod "6029d56a-5483-4a75-be08-bb6be38b21ac" (UID: "6029d56a-5483-4a75-be08-bb6be38b21ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:14:48.335084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.335054 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6029d56a-5483-4a75-be08-bb6be38b21ac-kube-api-access-24g8c" pod "6029d56a-5483-4a75-be08-bb6be38b21ac" (UID: "6029d56a-5483-4a75-be08-bb6be38b21ac"). InnerVolumeSpecName "kube-api-access-24g8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:48.434576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.434547 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:48.434576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.434574 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6029d56a-5483-4a75-be08-bb6be38b21ac-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:48.434771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.434584 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24g8c\" (UniqueName: \"kubernetes.io/projected/6029d56a-5483-4a75-be08-bb6be38b21ac-kube-api-access-24g8c\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:48.556466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:48.556401 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:49.065168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:49.065138 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" Nov 02 14:14:49.065664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:49.065173 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf" event={"ID":"6029d56a-5483-4a75-be08-bb6be38b21ac","Type":"ContainerDied","Data":"435eab6bb14ac917dbc15cc30dfcf7c5c88f7c0f049ed8f70b26933860143ac8"} Nov 02 14:14:49.065664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:49.065222 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="435eab6bb14ac917dbc15cc30dfcf7c5c88f7c0f049ed8f70b26933860143ac8" Nov 02 14:14:51.283418 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:14:51.283375 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:14:55.018789 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.018737 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj"] Nov 02 14:14:55.019601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.019576 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6029d56a-5483-4a75-be08-bb6be38b21ac" containerName="revoke-keys" Nov 02 14:14:55.019698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.019604 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6029d56a-5483-4a75-be08-bb6be38b21ac" containerName="revoke-keys" Nov 02 14:14:55.019848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.019827 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6029d56a-5483-4a75-be08-bb6be38b21ac" containerName="revoke-keys" Nov 02 14:14:55.024354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.024333 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.043193 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.043119 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj"] Nov 02 14:14:55.049491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.049471 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:14:55.049636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.049567 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:14:55.144819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.144768 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvqnv\" (UniqueName: \"kubernetes.io/projected/ed539f16-e6dc-4389-a554-6a36eb55b0a8-kube-api-access-rvqnv\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.144990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.144970 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.145083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.145067 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.246467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.246439 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvqnv\" (UniqueName: \"kubernetes.io/projected/ed539f16-e6dc-4389-a554-6a36eb55b0a8-kube-api-access-rvqnv\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.246620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.246525 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.246620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.246565 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.247148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.247129 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.248950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.248930 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.271938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.271882 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvqnv\" (UniqueName: \"kubernetes.io/projected/ed539f16-e6dc-4389-a554-6a36eb55b0a8-kube-api-access-rvqnv\") pod \"maas-api-revoke-sub-f4114ef0-fp7nj\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.344705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.344680 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:55.659179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:55.659140 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj"] Nov 02 14:14:55.659340 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:14:55.659319 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded539f16_e6dc_4389_a554_6a36eb55b0a8.slice/crio-949d8f22fe0d320611af626011b1102b9eab26515042c1e8758e4624f0da8244 WatchSource:0}: Error finding container 949d8f22fe0d320611af626011b1102b9eab26515042c1e8758e4624f0da8244: Status 404 returned error can't find the container with id 949d8f22fe0d320611af626011b1102b9eab26515042c1e8758e4624f0da8244 Nov 02 14:14:56.166886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:56.166850 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" event={"ID":"ed539f16-e6dc-4389-a554-6a36eb55b0a8","Type":"ContainerStarted","Data":"c062f3ead6e82095cba0260fbba3bf58f4427b48315b8a0c4e139df583cdf455"} Nov 02 14:14:56.167522 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:56.166911 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" event={"ID":"ed539f16-e6dc-4389-a554-6a36eb55b0a8","Type":"ContainerStarted","Data":"949d8f22fe0d320611af626011b1102b9eab26515042c1e8758e4624f0da8244"} Nov 02 14:14:56.214086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:56.214035 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" podStartSLOduration=2.214021571 podStartE2EDuration="2.214021571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:56.21132984 +0000 UTC m=+1061.248434774" watchObservedRunningTime="2026-11-02 14:14:56.214021571 +0000 UTC m=+1061.251126502" Nov 02 14:14:57.176326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:57.176106 4431 generic.go:356] "Generic (PLEG): container finished" podID="ed539f16-e6dc-4389-a554-6a36eb55b0a8" containerID="c062f3ead6e82095cba0260fbba3bf58f4427b48315b8a0c4e139df583cdf455" exitCode=0 Nov 02 14:14:57.176326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:57.176190 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" event={"ID":"ed539f16-e6dc-4389-a554-6a36eb55b0a8","Type":"ContainerDied","Data":"c062f3ead6e82095cba0260fbba3bf58f4427b48315b8a0c4e139df583cdf455"} Nov 02 14:14:57.556358 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:57.556275 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:57.568550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:57.568525 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:58.195191 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.195161 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:14:58.364314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.364287 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:58.493977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.493891 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed539f16-e6dc-4389-a554-6a36eb55b0a8-kube-api-access-rvqnv\" (UniqueName: \"kubernetes.io/projected/ed539f16-e6dc-4389-a554-6a36eb55b0a8-kube-api-access-rvqnv\") pod \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " Nov 02 14:14:58.494145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.493975 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-xks-ca\") pod \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " Nov 02 14:14:58.494145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.494093 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-openshift-ca\") pod \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\" (UID: \"ed539f16-e6dc-4389-a554-6a36eb55b0a8\") " Nov 02 14:14:58.494423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.494390 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-openshift-ca" pod "ed539f16-e6dc-4389-a554-6a36eb55b0a8" (UID: "ed539f16-e6dc-4389-a554-6a36eb55b0a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:14:58.494574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.494540 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:58.496261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.496232 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-xks-ca" pod "ed539f16-e6dc-4389-a554-6a36eb55b0a8" (UID: "ed539f16-e6dc-4389-a554-6a36eb55b0a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:14:58.496380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.496319 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed539f16-e6dc-4389-a554-6a36eb55b0a8-kube-api-access-rvqnv" pod "ed539f16-e6dc-4389-a554-6a36eb55b0a8" (UID: "ed539f16-e6dc-4389-a554-6a36eb55b0a8"). InnerVolumeSpecName "kube-api-access-rvqnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:14:58.595868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.595834 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvqnv\" (UniqueName: \"kubernetes.io/projected/ed539f16-e6dc-4389-a554-6a36eb55b0a8-kube-api-access-rvqnv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:58.595868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:58.595864 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed539f16-e6dc-4389-a554-6a36eb55b0a8-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:14:59.193208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:59.193177 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" Nov 02 14:14:59.193361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:59.193200 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj" event={"ID":"ed539f16-e6dc-4389-a554-6a36eb55b0a8","Type":"ContainerDied","Data":"949d8f22fe0d320611af626011b1102b9eab26515042c1e8758e4624f0da8244"} Nov 02 14:14:59.193361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:14:59.193251 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="949d8f22fe0d320611af626011b1102b9eab26515042c1e8758e4624f0da8244" Nov 02 14:15:00.157040 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.157009 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p"] Nov 02 14:15:00.157517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.157504 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ed539f16-e6dc-4389-a554-6a36eb55b0a8" containerName="revoke-keys" Nov 02 14:15:00.157566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.157517 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed539f16-e6dc-4389-a554-6a36eb55b0a8" containerName="revoke-keys" Nov 02 14:15:00.157634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.157625 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed539f16-e6dc-4389-a554-6a36eb55b0a8" containerName="revoke-keys" Nov 02 14:15:00.161422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.161407 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:00.173017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.172994 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9"] Nov 02 14:15:00.179293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.179273 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk"] Nov 02 14:15:00.179405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.179383 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:00.181636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.181618 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:15:00.183289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.183273 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62"] Nov 02 14:15:00.183387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.183370 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:00.191532 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.191512 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x"] Nov 02 14:15:00.191645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.191623 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:00.213817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.196472 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p"] Nov 02 14:15:00.213817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.196624 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:00.213817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.210748 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9"] Nov 02 14:15:00.217725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.217698 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk"] Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.222908 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4zs\" (UniqueName: \"kubernetes.io/projected/dc6cb207-255f-4137-a9a9-88802df4aaa1-kube-api-access-cs4zs\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9\" (UID: \"dc6cb207-255f-4137-a9a9-88802df4aaa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.222962 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/710e73b7-1eaa-471c-b239-f23b819dfa93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk\" (UID: \"710e73b7-1eaa-471c-b239-f23b819dfa93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.222989 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22decce4-c69a-4daa-a2c7-14dcce02f250-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x\" (UID: \"22decce4-c69a-4daa-a2c7-14dcce02f250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.223076 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swg5v\" (UniqueName: \"kubernetes.io/projected/e2cce962-7c39-4aa6-806b-a81d805ce090-kube-api-access-swg5v\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62\" (UID: \"e2cce962-7c39-4aa6-806b-a81d805ce090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.223113 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cce962-7c39-4aa6-806b-a81d805ce090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62\" (UID: \"e2cce962-7c39-4aa6-806b-a81d805ce090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.223153 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwkvb\" (UniqueName: \"kubernetes.io/projected/22decce4-c69a-4daa-a2c7-14dcce02f250-kube-api-access-zwkvb\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x\" (UID: \"22decce4-c69a-4daa-a2c7-14dcce02f250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.223544 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8pr\" (UniqueName: \"kubernetes.io/projected/a4177388-b90c-46e8-8654-6c9393fd2622-kube-api-access-tt8pr\") pod \"maas-api-key-cleanup-29893815-29r2p\" (UID: \"a4177388-b90c-46e8-8654-6c9393fd2622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.223633 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cb207-255f-4137-a9a9-88802df4aaa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9\" (UID: \"dc6cb207-255f-4137-a9a9-88802df4aaa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.223667 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59zhb\" (UniqueName: \"kubernetes.io/projected/710e73b7-1eaa-471c-b239-f23b819dfa93-kube-api-access-59zhb\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk\" (UID: \"710e73b7-1eaa-471c-b239-f23b819dfa93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.223733 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4177388-b90c-46e8-8654-6c9393fd2622-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-29r2p\" (UID: \"a4177388-b90c-46e8-8654-6c9393fd2622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:00.226398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.223976 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z"] Nov 02 14:15:00.232490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.229701 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:00.232490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.230761 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x"] Nov 02 14:15:00.242348 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.242327 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62"] Nov 02 14:15:00.248546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.248525 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z"] Nov 02 14:15:00.324702 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.324675 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4zs\" (UniqueName: \"kubernetes.io/projected/dc6cb207-255f-4137-a9a9-88802df4aaa1-kube-api-access-cs4zs\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9\" (UID: \"dc6cb207-255f-4137-a9a9-88802df4aaa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:00.324877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.324721 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/710e73b7-1eaa-471c-b239-f23b819dfa93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk\" (UID: \"710e73b7-1eaa-471c-b239-f23b819dfa93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:00.324877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.324742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22decce4-c69a-4daa-a2c7-14dcce02f250-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x\" (UID: \"22decce4-c69a-4daa-a2c7-14dcce02f250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:00.324877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.324838 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swg5v\" (UniqueName: \"kubernetes.io/projected/e2cce962-7c39-4aa6-806b-a81d805ce090-kube-api-access-swg5v\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62\" (UID: \"e2cce962-7c39-4aa6-806b-a81d805ce090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:00.324877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.324869 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cce962-7c39-4aa6-806b-a81d805ce090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62\" (UID: \"e2cce962-7c39-4aa6-806b-a81d805ce090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:00.325100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.324899 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwkvb\" (UniqueName: \"kubernetes.io/projected/22decce4-c69a-4daa-a2c7-14dcce02f250-kube-api-access-zwkvb\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x\" (UID: \"22decce4-c69a-4daa-a2c7-14dcce02f250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:00.325100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.324945 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8pr\" (UniqueName: \"kubernetes.io/projected/a4177388-b90c-46e8-8654-6c9393fd2622-kube-api-access-tt8pr\") pod \"maas-api-key-cleanup-29893815-29r2p\" (UID: \"a4177388-b90c-46e8-8654-6c9393fd2622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:00.325100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325022 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cb207-255f-4137-a9a9-88802df4aaa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9\" (UID: \"dc6cb207-255f-4137-a9a9-88802df4aaa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:00.325100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325054 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc92d0-8b09-4876-95dc-16113a80e34a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z\" (UID: \"8efc92d0-8b09-4876-95dc-16113a80e34a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:00.325100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325083 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zhb\" (UniqueName: \"kubernetes.io/projected/710e73b7-1eaa-471c-b239-f23b819dfa93-kube-api-access-59zhb\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk\" (UID: \"710e73b7-1eaa-471c-b239-f23b819dfa93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:00.325366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325150 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4177388-b90c-46e8-8654-6c9393fd2622-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-29r2p\" (UID: \"a4177388-b90c-46e8-8654-6c9393fd2622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:00.325366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325182 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrgz\" (UniqueName: \"kubernetes.io/projected/8efc92d0-8b09-4876-95dc-16113a80e34a-kube-api-access-vsrgz\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z\" (UID: \"8efc92d0-8b09-4876-95dc-16113a80e34a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:00.325591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325566 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22decce4-c69a-4daa-a2c7-14dcce02f250-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x\" (UID: \"22decce4-c69a-4daa-a2c7-14dcce02f250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:00.325663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325611 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/710e73b7-1eaa-471c-b239-f23b819dfa93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk\" (UID: \"710e73b7-1eaa-471c-b239-f23b819dfa93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:00.325765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325749 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cb207-255f-4137-a9a9-88802df4aaa1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9\" (UID: \"dc6cb207-255f-4137-a9a9-88802df4aaa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:00.325839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325753 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cce962-7c39-4aa6-806b-a81d805ce090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62\" (UID: \"e2cce962-7c39-4aa6-806b-a81d805ce090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:00.325953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.325937 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4177388-b90c-46e8-8654-6c9393fd2622-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-29r2p\" (UID: \"a4177388-b90c-46e8-8654-6c9393fd2622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:00.350097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.350050 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4zs\" (UniqueName: \"kubernetes.io/projected/dc6cb207-255f-4137-a9a9-88802df4aaa1-kube-api-access-cs4zs\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9\" (UID: \"dc6cb207-255f-4137-a9a9-88802df4aaa1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:00.350236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.350122 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwkvb\" (UniqueName: \"kubernetes.io/projected/22decce4-c69a-4daa-a2c7-14dcce02f250-kube-api-access-zwkvb\") pod \"maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x\" (UID: \"22decce4-c69a-4daa-a2c7-14dcce02f250\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:00.350306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.350275 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8pr\" (UniqueName: \"kubernetes.io/projected/a4177388-b90c-46e8-8654-6c9393fd2622-kube-api-access-tt8pr\") pod \"maas-api-key-cleanup-29893815-29r2p\" (UID: \"a4177388-b90c-46e8-8654-6c9393fd2622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:00.350382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.350342 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swg5v\" (UniqueName: \"kubernetes.io/projected/e2cce962-7c39-4aa6-806b-a81d805ce090-kube-api-access-swg5v\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62\" (UID: \"e2cce962-7c39-4aa6-806b-a81d805ce090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:00.350517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.350499 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zhb\" (UniqueName: \"kubernetes.io/projected/710e73b7-1eaa-471c-b239-f23b819dfa93-kube-api-access-59zhb\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk\" (UID: \"710e73b7-1eaa-471c-b239-f23b819dfa93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:00.426700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.426633 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc92d0-8b09-4876-95dc-16113a80e34a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z\" (UID: \"8efc92d0-8b09-4876-95dc-16113a80e34a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:00.426928 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.426695 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrgz\" (UniqueName: \"kubernetes.io/projected/8efc92d0-8b09-4876-95dc-16113a80e34a-kube-api-access-vsrgz\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z\" (UID: \"8efc92d0-8b09-4876-95dc-16113a80e34a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:00.427222 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.427202 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc92d0-8b09-4876-95dc-16113a80e34a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z\" (UID: \"8efc92d0-8b09-4876-95dc-16113a80e34a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:00.454023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.453995 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrgz\" (UniqueName: \"kubernetes.io/projected/8efc92d0-8b09-4876-95dc-16113a80e34a-kube-api-access-vsrgz\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z\" (UID: \"8efc92d0-8b09-4876-95dc-16113a80e34a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:00.476508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.476490 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:00.502466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.502443 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:00.519739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.519710 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:00.541643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.541614 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:00.562012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.561241 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:00.562012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.561592 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:00.820857 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:00.820817 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4177388_b90c_46e8_8654_6c9393fd2622.slice/crio-fa1196900cb3855eb897eb054f681631b294667e4530f2dd0ed030660436fb8c WatchSource:0}: Error finding container fa1196900cb3855eb897eb054f681631b294667e4530f2dd0ed030660436fb8c: Status 404 returned error can't find the container with id fa1196900cb3855eb897eb054f681631b294667e4530f2dd0ed030660436fb8c Nov 02 14:15:00.822246 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.822215 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p"] Nov 02 14:15:00.824941 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:00.824915 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6cb207_255f_4137_a9a9_88802df4aaa1.slice/crio-ba30a68f87e315e140e722ba434bb5d269abacc67ff8e18a6d967abab33626b1 WatchSource:0}: Error finding container ba30a68f87e315e140e722ba434bb5d269abacc67ff8e18a6d967abab33626b1: Status 404 returned error can't find the container with id ba30a68f87e315e140e722ba434bb5d269abacc67ff8e18a6d967abab33626b1 Nov 02 14:15:00.829418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.829390 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9"] Nov 02 14:15:00.874299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.874277 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk"] Nov 02 14:15:00.876079 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:00.876049 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710e73b7_1eaa_471c_b239_f23b819dfa93.slice/crio-afad16b452a5e2c240540ef6902958910f77b200ada38154147e5041c34d51bb WatchSource:0}: Error finding container afad16b452a5e2c240540ef6902958910f77b200ada38154147e5041c34d51bb: Status 404 returned error can't find the container with id afad16b452a5e2c240540ef6902958910f77b200ada38154147e5041c34d51bb Nov 02 14:15:00.920552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.920472 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62"] Nov 02 14:15:00.927324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.927296 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z"] Nov 02 14:15:00.937102 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:00.937076 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x"] Nov 02 14:15:00.981692 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:00.981659 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cce962_7c39_4aa6_806b_a81d805ce090.slice/crio-52355cb3b6dc3ed3704dcf9c368348a4a376da49f86fb5bff1713764a4ee96f3 WatchSource:0}: Error finding container 52355cb3b6dc3ed3704dcf9c368348a4a376da49f86fb5bff1713764a4ee96f3: Status 404 returned error can't find the container with id 52355cb3b6dc3ed3704dcf9c368348a4a376da49f86fb5bff1713764a4ee96f3 Nov 02 14:15:00.982366 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:00.982343 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22decce4_c69a_4daa_a2c7_14dcce02f250.slice/crio-3793e164e3e3c3f2699f734a083c1bac2f94f680cb758f7fc6b5a9b8906068df WatchSource:0}: Error finding container 3793e164e3e3c3f2699f734a083c1bac2f94f680cb758f7fc6b5a9b8906068df: Status 404 returned error can't find the container with id 3793e164e3e3c3f2699f734a083c1bac2f94f680cb758f7fc6b5a9b8906068df Nov 02 14:15:01.222503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:01.222469 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" event={"ID":"8efc92d0-8b09-4876-95dc-16113a80e34a","Type":"ContainerStarted","Data":"e5e5979b64558cea5ed4bba0278aa4fee7c2a91e83d57eb34c4a02f9c21c3e90"} Nov 02 14:15:01.224336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:01.224305 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" event={"ID":"dc6cb207-255f-4137-a9a9-88802df4aaa1","Type":"ContainerStarted","Data":"2f48cd7e839ebddda957925f12060cbe3ce1f97201d96b4c93a0e156e37592d0"} Nov 02 14:15:01.224482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:01.224361 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" event={"ID":"dc6cb207-255f-4137-a9a9-88802df4aaa1","Type":"ContainerStarted","Data":"ba30a68f87e315e140e722ba434bb5d269abacc67ff8e18a6d967abab33626b1"} Nov 02 14:15:01.225831 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:01.225804 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" event={"ID":"e2cce962-7c39-4aa6-806b-a81d805ce090","Type":"ContainerStarted","Data":"52355cb3b6dc3ed3704dcf9c368348a4a376da49f86fb5bff1713764a4ee96f3"} Nov 02 14:15:01.227197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:01.227169 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" event={"ID":"a4177388-b90c-46e8-8654-6c9393fd2622","Type":"ContainerStarted","Data":"fa1196900cb3855eb897eb054f681631b294667e4530f2dd0ed030660436fb8c"} Nov 02 14:15:01.228651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:01.228576 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" event={"ID":"22decce4-c69a-4daa-a2c7-14dcce02f250","Type":"ContainerStarted","Data":"3793e164e3e3c3f2699f734a083c1bac2f94f680cb758f7fc6b5a9b8906068df"} Nov 02 14:15:01.230022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:01.230002 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" event={"ID":"710e73b7-1eaa-471c-b239-f23b819dfa93","Type":"ContainerStarted","Data":"afad16b452a5e2c240540ef6902958910f77b200ada38154147e5041c34d51bb"} Nov 02 14:15:01.267949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:01.267899 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" podStartSLOduration=1.267883035 podStartE2EDuration="1.267883035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:01.267131689 +0000 UTC m=+1066.304236622" watchObservedRunningTime="2026-11-02 14:15:01.267883035 +0000 UTC m=+1066.304987969" Nov 02 14:15:02.244180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.244145 4431 generic.go:356] "Generic (PLEG): container finished" podID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerID="94855fffa0d4d9c1b0448196b09b892387f7657ef466387dda4f7e699f3baea9" exitCode=60 Nov 02 14:15:02.244686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.244321 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" event={"ID":"710e73b7-1eaa-471c-b239-f23b819dfa93","Type":"ContainerDied","Data":"94855fffa0d4d9c1b0448196b09b892387f7657ef466387dda4f7e699f3baea9"} Nov 02 14:15:02.244686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.244512 4431 scope.go:122] "RemoveContainer" containerID="94855fffa0d4d9c1b0448196b09b892387f7657ef466387dda4f7e699f3baea9" Nov 02 14:15:02.247727 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.247701 4431 generic.go:356] "Generic (PLEG): container finished" podID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerID="982e1b6061858f9568b93725c49f51a847d16705ca5dfda9a3a7e684946902b8" exitCode=60 Nov 02 14:15:02.247864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.247812 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" event={"ID":"8efc92d0-8b09-4876-95dc-16113a80e34a","Type":"ContainerDied","Data":"982e1b6061858f9568b93725c49f51a847d16705ca5dfda9a3a7e684946902b8"} Nov 02 14:15:02.248070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.248054 4431 scope.go:122] "RemoveContainer" containerID="982e1b6061858f9568b93725c49f51a847d16705ca5dfda9a3a7e684946902b8" Nov 02 14:15:02.249540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.249512 4431 generic.go:356] "Generic (PLEG): container finished" podID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerID="2f48cd7e839ebddda957925f12060cbe3ce1f97201d96b4c93a0e156e37592d0" exitCode=60 Nov 02 14:15:02.249624 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.249594 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" event={"ID":"dc6cb207-255f-4137-a9a9-88802df4aaa1","Type":"ContainerDied","Data":"2f48cd7e839ebddda957925f12060cbe3ce1f97201d96b4c93a0e156e37592d0"} Nov 02 14:15:02.249858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.249840 4431 scope.go:122] "RemoveContainer" containerID="2f48cd7e839ebddda957925f12060cbe3ce1f97201d96b4c93a0e156e37592d0" Nov 02 14:15:02.251103 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.251085 4431 generic.go:356] "Generic (PLEG): container finished" podID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerID="11a94ceacffa52c0b867577a9b6b8bf8c97fe414680977c5e0ca2693b954eafa" exitCode=60 Nov 02 14:15:02.251189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.251158 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" event={"ID":"e2cce962-7c39-4aa6-806b-a81d805ce090","Type":"ContainerDied","Data":"11a94ceacffa52c0b867577a9b6b8bf8c97fe414680977c5e0ca2693b954eafa"} Nov 02 14:15:02.251323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.251311 4431 scope.go:122] "RemoveContainer" containerID="11a94ceacffa52c0b867577a9b6b8bf8c97fe414680977c5e0ca2693b954eafa" Nov 02 14:15:02.252768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.252750 4431 generic.go:356] "Generic (PLEG): container finished" podID="a4177388-b90c-46e8-8654-6c9393fd2622" containerID="1a72618a12e42e0ca1e67d52b61c87da3ed987cb03c94a942d417f3ad26a699f" exitCode=60 Nov 02 14:15:02.252921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.252840 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" event={"ID":"a4177388-b90c-46e8-8654-6c9393fd2622","Type":"ContainerDied","Data":"1a72618a12e42e0ca1e67d52b61c87da3ed987cb03c94a942d417f3ad26a699f"} Nov 02 14:15:02.253102 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.253084 4431 scope.go:122] "RemoveContainer" containerID="1a72618a12e42e0ca1e67d52b61c87da3ed987cb03c94a942d417f3ad26a699f" Nov 02 14:15:02.254845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.254825 4431 generic.go:356] "Generic (PLEG): container finished" podID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerID="fafbd53eaf881ccbe3cda8a6dd39c31ff9739661c5297fb999f6657e00045d7d" exitCode=60 Nov 02 14:15:02.254945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.254912 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" event={"ID":"22decce4-c69a-4daa-a2c7-14dcce02f250","Type":"ContainerDied","Data":"fafbd53eaf881ccbe3cda8a6dd39c31ff9739661c5297fb999f6657e00045d7d"} Nov 02 14:15:02.255047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:02.255035 4431 scope.go:122] "RemoveContainer" containerID="fafbd53eaf881ccbe3cda8a6dd39c31ff9739661c5297fb999f6657e00045d7d" Nov 02 14:15:03.264860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.264817 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" event={"ID":"710e73b7-1eaa-471c-b239-f23b819dfa93","Type":"ContainerStarted","Data":"63b432cf8a7c937c6b88f7168cc74bd907242acdf19f14fecfa6bd3d8fb507af"} Nov 02 14:15:03.266985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.266957 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" event={"ID":"8efc92d0-8b09-4876-95dc-16113a80e34a","Type":"ContainerStarted","Data":"d0146ebf9cd5ae9aa7ea45eafc8779783b5ed5e464f8cb6c851085bd034183e6"} Nov 02 14:15:03.268866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.268843 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" event={"ID":"dc6cb207-255f-4137-a9a9-88802df4aaa1","Type":"ContainerStarted","Data":"aeafea7eb95ea1d88126df2ffb6b3b96554cf12b97f7e2626c9c8367b0e230f4"} Nov 02 14:15:03.271111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.271082 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" event={"ID":"e2cce962-7c39-4aa6-806b-a81d805ce090","Type":"ContainerStarted","Data":"026861b9770e5ca47766c7620d4c85a5f79f5112c5e41f00e064e70524cb1b4b"} Nov 02 14:15:03.274389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.274365 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" event={"ID":"a4177388-b90c-46e8-8654-6c9393fd2622","Type":"ContainerStarted","Data":"20df9a984a2fe509d9d49a37706699bc2703d75a1eb9dba49a5db902f0459e18"} Nov 02 14:15:03.278088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.278068 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" event={"ID":"22decce4-c69a-4daa-a2c7-14dcce02f250","Type":"ContainerStarted","Data":"f6973bde21eb7fa63cdfe6c8a7433bc788a30ec2e9e3c107856ec158351f863a"} Nov 02 14:15:03.317686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.317639 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" podStartSLOduration=3.317626365 podStartE2EDuration="3.317626365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.313085736 +0000 UTC m=+1068.350190672" watchObservedRunningTime="2026-11-02 14:15:03.317626365 +0000 UTC m=+1068.354731297" Nov 02 14:15:03.361564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.361522 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" podStartSLOduration=3.361510401 podStartE2EDuration="3.361510401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.359115767 +0000 UTC m=+1068.396220715" watchObservedRunningTime="2026-11-02 14:15:03.361510401 +0000 UTC m=+1068.398615332" Nov 02 14:15:03.405061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.405017 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" podStartSLOduration=3.40500452 podStartE2EDuration="3.40500452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.403022352 +0000 UTC m=+1068.440127284" watchObservedRunningTime="2026-11-02 14:15:03.40500452 +0000 UTC m=+1068.442109452" Nov 02 14:15:03.449407 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.449351 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" podStartSLOduration=3.449333272 podStartE2EDuration="3.449333272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.445542677 +0000 UTC m=+1068.482647622" watchObservedRunningTime="2026-11-02 14:15:03.449333272 +0000 UTC m=+1068.486438208" Nov 02 14:15:03.492645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:03.492596 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" podStartSLOduration=3.492582699 podStartE2EDuration="3.492582699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.487883811 +0000 UTC m=+1068.524988735" watchObservedRunningTime="2026-11-02 14:15:03.492582699 +0000 UTC m=+1068.529687680" Nov 02 14:15:04.288501 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.288460 4431 generic.go:356] "Generic (PLEG): container finished" podID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerID="63b432cf8a7c937c6b88f7168cc74bd907242acdf19f14fecfa6bd3d8fb507af" exitCode=60 Nov 02 14:15:04.288977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.288519 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" event={"ID":"710e73b7-1eaa-471c-b239-f23b819dfa93","Type":"ContainerDied","Data":"63b432cf8a7c937c6b88f7168cc74bd907242acdf19f14fecfa6bd3d8fb507af"} Nov 02 14:15:04.288977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.288598 4431 scope.go:122] "RemoveContainer" containerID="94855fffa0d4d9c1b0448196b09b892387f7657ef466387dda4f7e699f3baea9" Nov 02 14:15:04.288977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.288883 4431 scope.go:122] "RemoveContainer" containerID="63b432cf8a7c937c6b88f7168cc74bd907242acdf19f14fecfa6bd3d8fb507af" Nov 02 14:15:04.289226 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:04.289203 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk_odh-ai-gateway-infra(710e73b7-1eaa-471c-b239-f23b819dfa93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" Nov 02 14:15:04.290854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.290832 4431 generic.go:356] "Generic (PLEG): container finished" podID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerID="d0146ebf9cd5ae9aa7ea45eafc8779783b5ed5e464f8cb6c851085bd034183e6" exitCode=60 Nov 02 14:15:04.290963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.290901 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" event={"ID":"8efc92d0-8b09-4876-95dc-16113a80e34a","Type":"ContainerDied","Data":"d0146ebf9cd5ae9aa7ea45eafc8779783b5ed5e464f8cb6c851085bd034183e6"} Nov 02 14:15:04.291136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.291122 4431 scope.go:122] "RemoveContainer" containerID="d0146ebf9cd5ae9aa7ea45eafc8779783b5ed5e464f8cb6c851085bd034183e6" Nov 02 14:15:04.291379 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:04.291359 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z_odh-ai-gateway-infra(8efc92d0-8b09-4876-95dc-16113a80e34a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" Nov 02 14:15:04.292875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.292848 4431 generic.go:356] "Generic (PLEG): container finished" podID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerID="aeafea7eb95ea1d88126df2ffb6b3b96554cf12b97f7e2626c9c8367b0e230f4" exitCode=60 Nov 02 14:15:04.292962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.292937 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" event={"ID":"dc6cb207-255f-4137-a9a9-88802df4aaa1","Type":"ContainerDied","Data":"aeafea7eb95ea1d88126df2ffb6b3b96554cf12b97f7e2626c9c8367b0e230f4"} Nov 02 14:15:04.293158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.293141 4431 scope.go:122] "RemoveContainer" containerID="aeafea7eb95ea1d88126df2ffb6b3b96554cf12b97f7e2626c9c8367b0e230f4" Nov 02 14:15:04.293410 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:04.293384 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9_odh-ai-gateway-infra(dc6cb207-255f-4137-a9a9-88802df4aaa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" Nov 02 14:15:04.294945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.294925 4431 generic.go:356] "Generic (PLEG): container finished" podID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerID="026861b9770e5ca47766c7620d4c85a5f79f5112c5e41f00e064e70524cb1b4b" exitCode=60 Nov 02 14:15:04.295010 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.294988 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" event={"ID":"e2cce962-7c39-4aa6-806b-a81d805ce090","Type":"ContainerDied","Data":"026861b9770e5ca47766c7620d4c85a5f79f5112c5e41f00e064e70524cb1b4b"} Nov 02 14:15:04.295221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.295207 4431 scope.go:122] "RemoveContainer" containerID="026861b9770e5ca47766c7620d4c85a5f79f5112c5e41f00e064e70524cb1b4b" Nov 02 14:15:04.295432 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:04.295411 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62_odh-ai-gateway-infra(e2cce962-7c39-4aa6-806b-a81d805ce090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" Nov 02 14:15:04.297008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.296989 4431 generic.go:356] "Generic (PLEG): container finished" podID="a4177388-b90c-46e8-8654-6c9393fd2622" containerID="20df9a984a2fe509d9d49a37706699bc2703d75a1eb9dba49a5db902f0459e18" exitCode=60 Nov 02 14:15:04.297111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.297070 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" event={"ID":"a4177388-b90c-46e8-8654-6c9393fd2622","Type":"ContainerDied","Data":"20df9a984a2fe509d9d49a37706699bc2703d75a1eb9dba49a5db902f0459e18"} Nov 02 14:15:04.297241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.297223 4431 scope.go:122] "RemoveContainer" containerID="20df9a984a2fe509d9d49a37706699bc2703d75a1eb9dba49a5db902f0459e18" Nov 02 14:15:04.297440 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:04.297422 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-29r2p_odh-ai-gateway-infra(a4177388-b90c-46e8-8654-6c9393fd2622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" Nov 02 14:15:04.298939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.298920 4431 generic.go:356] "Generic (PLEG): container finished" podID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerID="f6973bde21eb7fa63cdfe6c8a7433bc788a30ec2e9e3c107856ec158351f863a" exitCode=60 Nov 02 14:15:04.299036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.298944 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" event={"ID":"22decce4-c69a-4daa-a2c7-14dcce02f250","Type":"ContainerDied","Data":"f6973bde21eb7fa63cdfe6c8a7433bc788a30ec2e9e3c107856ec158351f863a"} Nov 02 14:15:04.299144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.299130 4431 scope.go:122] "RemoveContainer" containerID="f6973bde21eb7fa63cdfe6c8a7433bc788a30ec2e9e3c107856ec158351f863a" Nov 02 14:15:04.299324 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:04.299307 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x_odh-ai-gateway-infra(22decce4-c69a-4daa-a2c7-14dcce02f250)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" Nov 02 14:15:04.302894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.302874 4431 scope.go:122] "RemoveContainer" containerID="982e1b6061858f9568b93725c49f51a847d16705ca5dfda9a3a7e684946902b8" Nov 02 14:15:04.316758 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.316739 4431 scope.go:122] "RemoveContainer" containerID="2f48cd7e839ebddda957925f12060cbe3ce1f97201d96b4c93a0e156e37592d0" Nov 02 14:15:04.329486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.329467 4431 scope.go:122] "RemoveContainer" containerID="11a94ceacffa52c0b867577a9b6b8bf8c97fe414680977c5e0ca2693b954eafa" Nov 02 14:15:04.342267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.342245 4431 scope.go:122] "RemoveContainer" containerID="1a72618a12e42e0ca1e67d52b61c87da3ed987cb03c94a942d417f3ad26a699f" Nov 02 14:15:04.354679 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:04.354663 4431 scope.go:122] "RemoveContainer" containerID="fafbd53eaf881ccbe3cda8a6dd39c31ff9739661c5297fb999f6657e00045d7d" Nov 02 14:15:05.316112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:05.315676 4431 scope.go:122] "RemoveContainer" containerID="f6973bde21eb7fa63cdfe6c8a7433bc788a30ec2e9e3c107856ec158351f863a" Nov 02 14:15:05.316756 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:05.316207 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x_odh-ai-gateway-infra(22decce4-c69a-4daa-a2c7-14dcce02f250)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" Nov 02 14:15:05.318170 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:05.318146 4431 scope.go:122] "RemoveContainer" containerID="63b432cf8a7c937c6b88f7168cc74bd907242acdf19f14fecfa6bd3d8fb507af" Nov 02 14:15:05.318499 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:05.318475 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk_odh-ai-gateway-infra(710e73b7-1eaa-471c-b239-f23b819dfa93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" Nov 02 14:15:05.320076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:05.320058 4431 scope.go:122] "RemoveContainer" containerID="d0146ebf9cd5ae9aa7ea45eafc8779783b5ed5e464f8cb6c851085bd034183e6" Nov 02 14:15:05.320306 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:05.320287 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z_odh-ai-gateway-infra(8efc92d0-8b09-4876-95dc-16113a80e34a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" Nov 02 14:15:05.321949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:05.321927 4431 scope.go:122] "RemoveContainer" containerID="aeafea7eb95ea1d88126df2ffb6b3b96554cf12b97f7e2626c9c8367b0e230f4" Nov 02 14:15:05.322249 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:05.322224 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9_odh-ai-gateway-infra(dc6cb207-255f-4137-a9a9-88802df4aaa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" Nov 02 14:15:05.323832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:05.323814 4431 scope.go:122] "RemoveContainer" containerID="026861b9770e5ca47766c7620d4c85a5f79f5112c5e41f00e064e70524cb1b4b" Nov 02 14:15:05.324022 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:05.324004 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62_odh-ai-gateway-infra(e2cce962-7c39-4aa6-806b-a81d805ce090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" Nov 02 14:15:05.325450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:05.325434 4431 scope.go:122] "RemoveContainer" containerID="20df9a984a2fe509d9d49a37706699bc2703d75a1eb9dba49a5db902f0459e18" Nov 02 14:15:05.325612 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:05.325588 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-29r2p_odh-ai-gateway-infra(a4177388-b90c-46e8-8654-6c9393fd2622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" Nov 02 14:15:11.902677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:11.902633 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:15:15.073630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:15.073587 4431 scope.go:122] "RemoveContainer" containerID="20df9a984a2fe509d9d49a37706699bc2703d75a1eb9dba49a5db902f0459e18" Nov 02 14:15:15.074117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:15.073740 4431 scope.go:122] "RemoveContainer" containerID="aeafea7eb95ea1d88126df2ffb6b3b96554cf12b97f7e2626c9c8367b0e230f4" Nov 02 14:15:15.423666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:15.423640 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" event={"ID":"dc6cb207-255f-4137-a9a9-88802df4aaa1","Type":"ContainerStarted","Data":"f002d617752781a1e47ff3ec1595f44f8bf5db81f78538df0ddf66457085bb7b"} Nov 02 14:15:16.069723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.069682 4431 scope.go:122] "RemoveContainer" containerID="f6973bde21eb7fa63cdfe6c8a7433bc788a30ec2e9e3c107856ec158351f863a" Nov 02 14:15:16.071174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.070038 4431 scope.go:122] "RemoveContainer" containerID="63b432cf8a7c937c6b88f7168cc74bd907242acdf19f14fecfa6bd3d8fb507af" Nov 02 14:15:16.071174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.070086 4431 scope.go:122] "RemoveContainer" containerID="026861b9770e5ca47766c7620d4c85a5f79f5112c5e41f00e064e70524cb1b4b" Nov 02 14:15:16.433153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.433119 4431 generic.go:356] "Generic (PLEG): container finished" podID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerID="f002d617752781a1e47ff3ec1595f44f8bf5db81f78538df0ddf66457085bb7b" exitCode=60 Nov 02 14:15:16.433500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.433167 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" event={"ID":"dc6cb207-255f-4137-a9a9-88802df4aaa1","Type":"ContainerDied","Data":"f002d617752781a1e47ff3ec1595f44f8bf5db81f78538df0ddf66457085bb7b"} Nov 02 14:15:16.433500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.433234 4431 scope.go:122] "RemoveContainer" containerID="aeafea7eb95ea1d88126df2ffb6b3b96554cf12b97f7e2626c9c8367b0e230f4" Nov 02 14:15:16.433621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.433605 4431 scope.go:122] "RemoveContainer" containerID="f002d617752781a1e47ff3ec1595f44f8bf5db81f78538df0ddf66457085bb7b" Nov 02 14:15:16.433918 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:16.433892 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9_odh-ai-gateway-infra(dc6cb207-255f-4137-a9a9-88802df4aaa1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" Nov 02 14:15:16.435354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.435333 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" event={"ID":"e2cce962-7c39-4aa6-806b-a81d805ce090","Type":"ContainerStarted","Data":"f129f3f4461848646c5ba22fdeabb0f21e3dabfd6535109313f5601a3ce9e188"} Nov 02 14:15:16.437052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.437033 4431 generic.go:356] "Generic (PLEG): container finished" podID="a4177388-b90c-46e8-8654-6c9393fd2622" containerID="8c6e8f2482a0e76df391dca8b827145606937c9df377bea593f9c3717655acbe" exitCode=60 Nov 02 14:15:16.437153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.437101 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" event={"ID":"a4177388-b90c-46e8-8654-6c9393fd2622","Type":"ContainerDied","Data":"8c6e8f2482a0e76df391dca8b827145606937c9df377bea593f9c3717655acbe"} Nov 02 14:15:16.437441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.437424 4431 scope.go:122] "RemoveContainer" containerID="8c6e8f2482a0e76df391dca8b827145606937c9df377bea593f9c3717655acbe" Nov 02 14:15:16.437692 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:16.437671 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-29r2p_odh-ai-gateway-infra(a4177388-b90c-46e8-8654-6c9393fd2622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" Nov 02 14:15:16.439197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.439177 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" event={"ID":"22decce4-c69a-4daa-a2c7-14dcce02f250","Type":"ContainerStarted","Data":"875db59c092605d8b3653105382911f72687bc9a30ef835813b8fb132be2d83c"} Nov 02 14:15:16.441082 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.441059 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" event={"ID":"710e73b7-1eaa-471c-b239-f23b819dfa93","Type":"ContainerStarted","Data":"01d2f6cc94326a37ca19fff88ed3099917b4b4116d6980a676d498317efb4d07"} Nov 02 14:15:16.449604 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.449587 4431 scope.go:122] "RemoveContainer" containerID="20df9a984a2fe509d9d49a37706699bc2703d75a1eb9dba49a5db902f0459e18" Nov 02 14:15:16.489354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:16.489332 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9"] Nov 02 14:15:17.456981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.456949 4431 generic.go:356] "Generic (PLEG): container finished" podID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerID="f129f3f4461848646c5ba22fdeabb0f21e3dabfd6535109313f5601a3ce9e188" exitCode=60 Nov 02 14:15:17.457378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.457024 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" event={"ID":"e2cce962-7c39-4aa6-806b-a81d805ce090","Type":"ContainerDied","Data":"f129f3f4461848646c5ba22fdeabb0f21e3dabfd6535109313f5601a3ce9e188"} Nov 02 14:15:17.457378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.457100 4431 scope.go:122] "RemoveContainer" containerID="026861b9770e5ca47766c7620d4c85a5f79f5112c5e41f00e064e70524cb1b4b" Nov 02 14:15:17.457502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.457485 4431 scope.go:122] "RemoveContainer" containerID="f129f3f4461848646c5ba22fdeabb0f21e3dabfd6535109313f5601a3ce9e188" Nov 02 14:15:17.457900 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:17.457878 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62_odh-ai-gateway-infra(e2cce962-7c39-4aa6-806b-a81d805ce090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" Nov 02 14:15:17.462909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.462884 4431 generic.go:356] "Generic (PLEG): container finished" podID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerID="875db59c092605d8b3653105382911f72687bc9a30ef835813b8fb132be2d83c" exitCode=60 Nov 02 14:15:17.463035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.462962 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" event={"ID":"22decce4-c69a-4daa-a2c7-14dcce02f250","Type":"ContainerDied","Data":"875db59c092605d8b3653105382911f72687bc9a30ef835813b8fb132be2d83c"} Nov 02 14:15:17.463355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.463336 4431 scope.go:122] "RemoveContainer" containerID="875db59c092605d8b3653105382911f72687bc9a30ef835813b8fb132be2d83c" Nov 02 14:15:17.463679 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:17.463651 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x_odh-ai-gateway-infra(22decce4-c69a-4daa-a2c7-14dcce02f250)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" Nov 02 14:15:17.465976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.465956 4431 generic.go:356] "Generic (PLEG): container finished" podID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerID="01d2f6cc94326a37ca19fff88ed3099917b4b4116d6980a676d498317efb4d07" exitCode=60 Nov 02 14:15:17.466085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.466022 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" event={"ID":"710e73b7-1eaa-471c-b239-f23b819dfa93","Type":"ContainerDied","Data":"01d2f6cc94326a37ca19fff88ed3099917b4b4116d6980a676d498317efb4d07"} Nov 02 14:15:17.466357 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.466341 4431 scope.go:122] "RemoveContainer" containerID="01d2f6cc94326a37ca19fff88ed3099917b4b4116d6980a676d498317efb4d07" Nov 02 14:15:17.466565 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:17.466548 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk_odh-ai-gateway-infra(710e73b7-1eaa-471c-b239-f23b819dfa93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" Nov 02 14:15:17.471329 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.471294 4431 scope.go:122] "RemoveContainer" containerID="f6973bde21eb7fa63cdfe6c8a7433bc788a30ec2e9e3c107856ec158351f863a" Nov 02 14:15:17.487728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.487705 4431 scope.go:122] "RemoveContainer" containerID="63b432cf8a7c937c6b88f7168cc74bd907242acdf19f14fecfa6bd3d8fb507af" Nov 02 14:15:17.542566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.541668 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x"] Nov 02 14:15:17.590316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.590283 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk"] Nov 02 14:15:17.635982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.634950 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62"] Nov 02 14:15:17.646966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.646938 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:17.678970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.678935 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6cb207-255f-4137-a9a9-88802df4aaa1-kube-api-access-cs4zs\" (UniqueName: \"kubernetes.io/projected/dc6cb207-255f-4137-a9a9-88802df4aaa1-kube-api-access-cs4zs\") pod \"dc6cb207-255f-4137-a9a9-88802df4aaa1\" (UID: \"dc6cb207-255f-4137-a9a9-88802df4aaa1\") " Nov 02 14:15:17.679340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.679306 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6cb207-255f-4137-a9a9-88802df4aaa1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cb207-255f-4137-a9a9-88802df4aaa1-maas-api-service-ca\") pod \"dc6cb207-255f-4137-a9a9-88802df4aaa1\" (UID: \"dc6cb207-255f-4137-a9a9-88802df4aaa1\") " Nov 02 14:15:17.680192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.680166 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6cb207-255f-4137-a9a9-88802df4aaa1-maas-api-service-ca" pod "dc6cb207-255f-4137-a9a9-88802df4aaa1" (UID: "dc6cb207-255f-4137-a9a9-88802df4aaa1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:17.683171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.683148 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6cb207-255f-4137-a9a9-88802df4aaa1-kube-api-access-cs4zs" pod "dc6cb207-255f-4137-a9a9-88802df4aaa1" (UID: "dc6cb207-255f-4137-a9a9-88802df4aaa1"). InnerVolumeSpecName "kube-api-access-cs4zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:17.684803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.684746 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p"] Nov 02 14:15:17.781347 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.781257 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4zs\" (UniqueName: \"kubernetes.io/projected/dc6cb207-255f-4137-a9a9-88802df4aaa1-kube-api-access-cs4zs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:17.781347 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.781284 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cb207-255f-4137-a9a9-88802df4aaa1-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:17.878256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.878233 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:17.983470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.983443 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4177388-b90c-46e8-8654-6c9393fd2622-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4177388-b90c-46e8-8654-6c9393fd2622-maas-api-service-ca\") pod \"a4177388-b90c-46e8-8654-6c9393fd2622\" (UID: \"a4177388-b90c-46e8-8654-6c9393fd2622\") " Nov 02 14:15:17.983602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.983578 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4177388-b90c-46e8-8654-6c9393fd2622-kube-api-access-tt8pr\" (UniqueName: \"kubernetes.io/projected/a4177388-b90c-46e8-8654-6c9393fd2622-kube-api-access-tt8pr\") pod \"a4177388-b90c-46e8-8654-6c9393fd2622\" (UID: \"a4177388-b90c-46e8-8654-6c9393fd2622\") " Nov 02 14:15:17.983834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.983814 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4177388-b90c-46e8-8654-6c9393fd2622-maas-api-service-ca" pod "a4177388-b90c-46e8-8654-6c9393fd2622" (UID: "a4177388-b90c-46e8-8654-6c9393fd2622"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:17.984040 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.984021 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4177388-b90c-46e8-8654-6c9393fd2622-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:17.985583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:17.985557 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4177388-b90c-46e8-8654-6c9393fd2622-kube-api-access-tt8pr" pod "a4177388-b90c-46e8-8654-6c9393fd2622" (UID: "a4177388-b90c-46e8-8654-6c9393fd2622"). InnerVolumeSpecName "kube-api-access-tt8pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:18.068972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.068911 4431 scope.go:122] "RemoveContainer" containerID="d0146ebf9cd5ae9aa7ea45eafc8779783b5ed5e464f8cb6c851085bd034183e6" Nov 02 14:15:18.085627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.084844 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8pr\" (UniqueName: \"kubernetes.io/projected/a4177388-b90c-46e8-8654-6c9393fd2622-kube-api-access-tt8pr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:18.479170 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.479147 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" event={"ID":"8efc92d0-8b09-4876-95dc-16113a80e34a","Type":"ContainerStarted","Data":"3e5aa2daa6968b1a8161afe6cae8641acdb889036ad0d8079a8bd47be1bc92ea"} Nov 02 14:15:18.480685 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.480661 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" Nov 02 14:15:18.480800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.480684 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9" event={"ID":"dc6cb207-255f-4137-a9a9-88802df4aaa1","Type":"ContainerDied","Data":"ba30a68f87e315e140e722ba434bb5d269abacc67ff8e18a6d967abab33626b1"} Nov 02 14:15:18.480800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.480739 4431 scope.go:122] "RemoveContainer" containerID="f002d617752781a1e47ff3ec1595f44f8bf5db81f78538df0ddf66457085bb7b" Nov 02 14:15:18.484334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.484313 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" Nov 02 14:15:18.484434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.484312 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p" event={"ID":"a4177388-b90c-46e8-8654-6c9393fd2622","Type":"ContainerDied","Data":"fa1196900cb3855eb897eb054f681631b294667e4530f2dd0ed030660436fb8c"} Nov 02 14:15:18.494594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.494560 4431 scope.go:122] "RemoveContainer" containerID="8c6e8f2482a0e76df391dca8b827145606937c9df377bea593f9c3717655acbe" Nov 02 14:15:18.658851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.658829 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:18.677240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.677207 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9"] Nov 02 14:15:18.681728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.681704 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893815-7vqj9"] Nov 02 14:15:18.692178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.692135 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22decce4-c69a-4daa-a2c7-14dcce02f250-kube-api-access-zwkvb\" (UniqueName: \"kubernetes.io/projected/22decce4-c69a-4daa-a2c7-14dcce02f250-kube-api-access-zwkvb\") pod \"22decce4-c69a-4daa-a2c7-14dcce02f250\" (UID: \"22decce4-c69a-4daa-a2c7-14dcce02f250\") " Nov 02 14:15:18.692322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.692305 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22decce4-c69a-4daa-a2c7-14dcce02f250-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22decce4-c69a-4daa-a2c7-14dcce02f250-maas-api-service-ca\") pod \"22decce4-c69a-4daa-a2c7-14dcce02f250\" (UID: \"22decce4-c69a-4daa-a2c7-14dcce02f250\") " Nov 02 14:15:18.692642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.692617 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22decce4-c69a-4daa-a2c7-14dcce02f250-maas-api-service-ca" pod "22decce4-c69a-4daa-a2c7-14dcce02f250" (UID: "22decce4-c69a-4daa-a2c7-14dcce02f250"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:18.692864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.692848 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22decce4-c69a-4daa-a2c7-14dcce02f250-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:18.701991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.701967 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22decce4-c69a-4daa-a2c7-14dcce02f250-kube-api-access-zwkvb" pod "22decce4-c69a-4daa-a2c7-14dcce02f250" (UID: "22decce4-c69a-4daa-a2c7-14dcce02f250"). InnerVolumeSpecName "kube-api-access-zwkvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:18.722160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.722140 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:18.726957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.726940 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:18.765521 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.765501 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p"] Nov 02 14:15:18.769686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.769665 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-29r2p"] Nov 02 14:15:18.793568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.793547 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710e73b7-1eaa-471c-b239-f23b819dfa93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/710e73b7-1eaa-471c-b239-f23b819dfa93-maas-api-service-ca\") pod \"710e73b7-1eaa-471c-b239-f23b819dfa93\" (UID: \"710e73b7-1eaa-471c-b239-f23b819dfa93\") " Nov 02 14:15:18.793669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.793605 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710e73b7-1eaa-471c-b239-f23b819dfa93-kube-api-access-59zhb\" (UniqueName: \"kubernetes.io/projected/710e73b7-1eaa-471c-b239-f23b819dfa93-kube-api-access-59zhb\") pod \"710e73b7-1eaa-471c-b239-f23b819dfa93\" (UID: \"710e73b7-1eaa-471c-b239-f23b819dfa93\") " Nov 02 14:15:18.793846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.793832 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2cce962-7c39-4aa6-806b-a81d805ce090-kube-api-access-swg5v\" (UniqueName: \"kubernetes.io/projected/e2cce962-7c39-4aa6-806b-a81d805ce090-kube-api-access-swg5v\") pod \"e2cce962-7c39-4aa6-806b-a81d805ce090\" (UID: \"e2cce962-7c39-4aa6-806b-a81d805ce090\") " Nov 02 14:15:18.793907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.793872 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cce962-7c39-4aa6-806b-a81d805ce090-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cce962-7c39-4aa6-806b-a81d805ce090-maas-api-service-ca\") pod \"e2cce962-7c39-4aa6-806b-a81d805ce090\" (UID: \"e2cce962-7c39-4aa6-806b-a81d805ce090\") " Nov 02 14:15:18.793978 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.793954 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710e73b7-1eaa-471c-b239-f23b819dfa93-maas-api-service-ca" pod "710e73b7-1eaa-471c-b239-f23b819dfa93" (UID: "710e73b7-1eaa-471c-b239-f23b819dfa93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:18.794323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.794293 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwkvb\" (UniqueName: \"kubernetes.io/projected/22decce4-c69a-4daa-a2c7-14dcce02f250-kube-api-access-zwkvb\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:18.794397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.794320 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/710e73b7-1eaa-471c-b239-f23b819dfa93-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:18.794397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.794341 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cce962-7c39-4aa6-806b-a81d805ce090-maas-api-service-ca" pod "e2cce962-7c39-4aa6-806b-a81d805ce090" (UID: "e2cce962-7c39-4aa6-806b-a81d805ce090"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:18.795956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.795931 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710e73b7-1eaa-471c-b239-f23b819dfa93-kube-api-access-59zhb" pod "710e73b7-1eaa-471c-b239-f23b819dfa93" (UID: "710e73b7-1eaa-471c-b239-f23b819dfa93"). InnerVolumeSpecName "kube-api-access-59zhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:18.796341 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.796317 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cce962-7c39-4aa6-806b-a81d805ce090-kube-api-access-swg5v" pod "e2cce962-7c39-4aa6-806b-a81d805ce090" (UID: "e2cce962-7c39-4aa6-806b-a81d805ce090"). InnerVolumeSpecName "kube-api-access-swg5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:18.895083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.895057 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swg5v\" (UniqueName: \"kubernetes.io/projected/e2cce962-7c39-4aa6-806b-a81d805ce090-kube-api-access-swg5v\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:18.895083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.895081 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cce962-7c39-4aa6-806b-a81d805ce090-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:18.895239 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:18.895091 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59zhb\" (UniqueName: \"kubernetes.io/projected/710e73b7-1eaa-471c-b239-f23b819dfa93-kube-api-access-59zhb\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:19.076054 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.076024 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" path="/var/lib/kubelet/pods/a4177388-b90c-46e8-8654-6c9393fd2622/volumes" Nov 02 14:15:19.076465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.076451 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" path="/var/lib/kubelet/pods/dc6cb207-255f-4137-a9a9-88802df4aaa1/volumes" Nov 02 14:15:19.497156 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.497116 4431 generic.go:356] "Generic (PLEG): container finished" podID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerID="3e5aa2daa6968b1a8161afe6cae8641acdb889036ad0d8079a8bd47be1bc92ea" exitCode=60 Nov 02 14:15:19.497637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.497266 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" event={"ID":"8efc92d0-8b09-4876-95dc-16113a80e34a","Type":"ContainerDied","Data":"3e5aa2daa6968b1a8161afe6cae8641acdb889036ad0d8079a8bd47be1bc92ea"} Nov 02 14:15:19.497637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.497361 4431 scope.go:122] "RemoveContainer" containerID="d0146ebf9cd5ae9aa7ea45eafc8779783b5ed5e464f8cb6c851085bd034183e6" Nov 02 14:15:19.497909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.497892 4431 scope.go:122] "RemoveContainer" containerID="3e5aa2daa6968b1a8161afe6cae8641acdb889036ad0d8079a8bd47be1bc92ea" Nov 02 14:15:19.498392 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:19.498364 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z_odh-ai-gateway-infra(8efc92d0-8b09-4876-95dc-16113a80e34a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" Nov 02 14:15:19.502460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.502440 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" Nov 02 14:15:19.502567 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.502442 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62" event={"ID":"e2cce962-7c39-4aa6-806b-a81d805ce090","Type":"ContainerDied","Data":"52355cb3b6dc3ed3704dcf9c368348a4a376da49f86fb5bff1713764a4ee96f3"} Nov 02 14:15:19.505217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.505198 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" Nov 02 14:15:19.505298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.505245 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x" event={"ID":"22decce4-c69a-4daa-a2c7-14dcce02f250","Type":"ContainerDied","Data":"3793e164e3e3c3f2699f734a083c1bac2f94f680cb758f7fc6b5a9b8906068df"} Nov 02 14:15:19.506621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.506597 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" event={"ID":"710e73b7-1eaa-471c-b239-f23b819dfa93","Type":"ContainerDied","Data":"afad16b452a5e2c240540ef6902958910f77b200ada38154147e5041c34d51bb"} Nov 02 14:15:19.506727 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.506606 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk" Nov 02 14:15:19.517165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.517105 4431 scope.go:122] "RemoveContainer" containerID="f129f3f4461848646c5ba22fdeabb0f21e3dabfd6535109313f5601a3ce9e188" Nov 02 14:15:19.528048 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.528030 4431 scope.go:122] "RemoveContainer" containerID="875db59c092605d8b3653105382911f72687bc9a30ef835813b8fb132be2d83c" Nov 02 14:15:19.536666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.536645 4431 scope.go:122] "RemoveContainer" containerID="01d2f6cc94326a37ca19fff88ed3099917b4b4116d6980a676d498317efb4d07" Nov 02 14:15:19.606379 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.606349 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x"] Nov 02 14:15:19.616168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.616139 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-555d1e-29893815-v628x"] Nov 02 14:15:19.644828 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.643644 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z"] Nov 02 14:15:19.669195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.669159 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk"] Nov 02 14:15:19.673265 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.673239 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893815-tnhfk"] Nov 02 14:15:19.716076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.716051 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62"] Nov 02 14:15:19.721887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:19.721865 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893815-dkw62"] Nov 02 14:15:20.660717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.660692 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:20.716422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.716391 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efc92d0-8b09-4876-95dc-16113a80e34a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc92d0-8b09-4876-95dc-16113a80e34a-maas-api-service-ca\") pod \"8efc92d0-8b09-4876-95dc-16113a80e34a\" (UID: \"8efc92d0-8b09-4876-95dc-16113a80e34a\") " Nov 02 14:15:20.716590 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.716448 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efc92d0-8b09-4876-95dc-16113a80e34a-kube-api-access-vsrgz\" (UniqueName: \"kubernetes.io/projected/8efc92d0-8b09-4876-95dc-16113a80e34a-kube-api-access-vsrgz\") pod \"8efc92d0-8b09-4876-95dc-16113a80e34a\" (UID: \"8efc92d0-8b09-4876-95dc-16113a80e34a\") " Nov 02 14:15:20.716863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.716766 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efc92d0-8b09-4876-95dc-16113a80e34a-maas-api-service-ca" pod "8efc92d0-8b09-4876-95dc-16113a80e34a" (UID: "8efc92d0-8b09-4876-95dc-16113a80e34a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:20.716950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.716928 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8efc92d0-8b09-4876-95dc-16113a80e34a-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:20.718626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.718597 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efc92d0-8b09-4876-95dc-16113a80e34a-kube-api-access-vsrgz" pod "8efc92d0-8b09-4876-95dc-16113a80e34a" (UID: "8efc92d0-8b09-4876-95dc-16113a80e34a"). InnerVolumeSpecName "kube-api-access-vsrgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:20.818130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.818058 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsrgz\" (UniqueName: \"kubernetes.io/projected/8efc92d0-8b09-4876-95dc-16113a80e34a-kube-api-access-vsrgz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:20.989719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.989685 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv"] Nov 02 14:15:20.990233 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990218 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.990282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990235 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.990282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990245 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:20.990282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990250 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:20.990282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990260 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:20.990282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990265 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:20.990282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990271 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:20.990282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990276 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990294 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990300 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990306 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990310 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990319 4431 cpu_manager.go:414] "containerMap: removing container" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990323 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990331 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990335 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990342 4431 cpu_manager.go:414] "containerMap: removing container" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990346 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990354 4431 cpu_manager.go:414] "containerMap: removing container" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990358 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990364 4431 cpu_manager.go:414] "containerMap: removing container" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990369 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990377 4431 cpu_manager.go:414] "containerMap: removing container" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990381 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990392 4431 cpu_manager.go:414] "containerMap: removing container" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990396 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990402 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990406 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990425 4431 cpu_manager.go:414] "containerMap: removing container" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:20.990505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990429 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990525 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990534 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990539 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990544 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990551 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990558 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990563 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990569 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990575 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990579 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990584 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990590 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990594 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990598 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990606 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" containerName="cleanup" Nov 02 14:15:20.991187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.990611 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:20.994493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:20.994475 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.010646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.010608 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:15:21.014667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.014645 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv"] Nov 02 14:15:21.020234 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.020210 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.020376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.020359 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.020456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.020414 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxw42\" (UniqueName: \"kubernetes.io/projected/9ed6bc10-83ff-481a-abe5-f0732aef7e05-kube-api-access-dxw42\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.078122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.078056 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22decce4-c69a-4daa-a2c7-14dcce02f250" path="/var/lib/kubelet/pods/22decce4-c69a-4daa-a2c7-14dcce02f250/volumes" Nov 02 14:15:21.078492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.078479 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" path="/var/lib/kubelet/pods/710e73b7-1eaa-471c-b239-f23b819dfa93/volumes" Nov 02 14:15:21.078907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.078893 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2cce962-7c39-4aa6-806b-a81d805ce090" path="/var/lib/kubelet/pods/e2cce962-7c39-4aa6-806b-a81d805ce090/volumes" Nov 02 14:15:21.121963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.121937 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.122108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.122064 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.122187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.122120 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxw42\" (UniqueName: \"kubernetes.io/projected/9ed6bc10-83ff-481a-abe5-f0732aef7e05-kube-api-access-dxw42\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.122708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.122687 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.124292 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.124271 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.146860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.146833 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxw42\" (UniqueName: \"kubernetes.io/projected/9ed6bc10-83ff-481a-abe5-f0732aef7e05-kube-api-access-dxw42\") pod \"maas-api-revoke-sub-8ad324bb-5zvhv\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.286861 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:21.286827 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:15:21.308283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.308256 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:21.526825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.526794 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" event={"ID":"8efc92d0-8b09-4876-95dc-16113a80e34a","Type":"ContainerDied","Data":"e5e5979b64558cea5ed4bba0278aa4fee7c2a91e83d57eb34c4a02f9c21c3e90"} Nov 02 14:15:21.526825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.526815 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z" Nov 02 14:15:21.527050 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.526853 4431 scope.go:122] "RemoveContainer" containerID="3e5aa2daa6968b1a8161afe6cae8641acdb889036ad0d8079a8bd47be1bc92ea" Nov 02 14:15:21.585258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.585229 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z"] Nov 02 14:15:21.591006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.590981 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893815-v4x7z"] Nov 02 14:15:21.624169 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:21.624125 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed6bc10_83ff_481a_abe5_f0732aef7e05.slice/crio-a1b4202b63a1ea81a309b3612b0271b2d55277423dd2fec9a316079d40efb25b WatchSource:0}: Error finding container a1b4202b63a1ea81a309b3612b0271b2d55277423dd2fec9a316079d40efb25b: Status 404 returned error can't find the container with id a1b4202b63a1ea81a309b3612b0271b2d55277423dd2fec9a316079d40efb25b Nov 02 14:15:21.624982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:21.624952 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv"] Nov 02 14:15:22.537378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:22.537299 4431 generic.go:356] "Generic (PLEG): container finished" podID="9ed6bc10-83ff-481a-abe5-f0732aef7e05" containerID="264575987ee30a853503b73c983859159d506b38ea2857d218eaf414a43d0b05" exitCode=0 Nov 02 14:15:22.537378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:22.537343 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" event={"ID":"9ed6bc10-83ff-481a-abe5-f0732aef7e05","Type":"ContainerDied","Data":"264575987ee30a853503b73c983859159d506b38ea2857d218eaf414a43d0b05"} Nov 02 14:15:22.537714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:22.537431 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" event={"ID":"9ed6bc10-83ff-481a-abe5-f0732aef7e05","Type":"ContainerStarted","Data":"a1b4202b63a1ea81a309b3612b0271b2d55277423dd2fec9a316079d40efb25b"} Nov 02 14:15:23.076327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.076292 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" path="/var/lib/kubelet/pods/8efc92d0-8b09-4876-95dc-16113a80e34a/volumes" Nov 02 14:15:23.716951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.716921 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:23.749251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.749224 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-xks-ca\") pod \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " Nov 02 14:15:23.749366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.749306 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed6bc10-83ff-481a-abe5-f0732aef7e05-kube-api-access-dxw42\" (UniqueName: \"kubernetes.io/projected/9ed6bc10-83ff-481a-abe5-f0732aef7e05-kube-api-access-dxw42\") pod \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " Nov 02 14:15:23.749366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.749346 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-openshift-ca\") pod \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\" (UID: \"9ed6bc10-83ff-481a-abe5-f0732aef7e05\") " Nov 02 14:15:23.749772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.749746 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-openshift-ca" pod "9ed6bc10-83ff-481a-abe5-f0732aef7e05" (UID: "9ed6bc10-83ff-481a-abe5-f0732aef7e05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:23.751214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.751187 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-xks-ca" pod "9ed6bc10-83ff-481a-abe5-f0732aef7e05" (UID: "9ed6bc10-83ff-481a-abe5-f0732aef7e05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:15:23.751371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.751348 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed6bc10-83ff-481a-abe5-f0732aef7e05-kube-api-access-dxw42" pod "9ed6bc10-83ff-481a-abe5-f0732aef7e05" (UID: "9ed6bc10-83ff-481a-abe5-f0732aef7e05"). InnerVolumeSpecName "kube-api-access-dxw42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:23.851147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.851072 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:23.851147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.851104 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed6bc10-83ff-481a-abe5-f0732aef7e05-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:23.851147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:23.851117 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxw42\" (UniqueName: \"kubernetes.io/projected/9ed6bc10-83ff-481a-abe5-f0732aef7e05-kube-api-access-dxw42\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:24.553849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:24.553825 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" Nov 02 14:15:24.553849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:24.553843 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv" event={"ID":"9ed6bc10-83ff-481a-abe5-f0732aef7e05","Type":"ContainerDied","Data":"a1b4202b63a1ea81a309b3612b0271b2d55277423dd2fec9a316079d40efb25b"} Nov 02 14:15:24.554026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:24.553891 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b4202b63a1ea81a309b3612b0271b2d55277423dd2fec9a316079d40efb25b" Nov 02 14:15:28.453589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:28.453542 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 14:15:28.525801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:28.521864 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:28.679898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:28.679855 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:28.996429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:28.996388 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:29.197801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:29.190852 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" podUID="5540857d-32be-4620-8c54-872b9a767f3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:29.224003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:29.223962 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:29.454395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:29.454338 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:15:29.455742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:29.455723 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:15:29.487414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:29.487354 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:15:30.332108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332075 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk"] Nov 02 14:15:30.332731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332712 4431 cpu_manager.go:414] "containerMap: removing container" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:30.332823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332732 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:30.332823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332743 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9ed6bc10-83ff-481a-abe5-f0732aef7e05" containerName="revoke-keys" Nov 02 14:15:30.332823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332748 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed6bc10-83ff-481a-abe5-f0732aef7e05" containerName="revoke-keys" Nov 02 14:15:30.332823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332769 4431 cpu_manager.go:414] "containerMap: removing container" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:30.332823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332789 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710e73b7-1eaa-471c-b239-f23b819dfa93" containerName="cleanup" Nov 02 14:15:30.332823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332808 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:30.332823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332817 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4177388-b90c-46e8-8654-6c9393fd2622" containerName="cleanup" Nov 02 14:15:30.333173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332944 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed6bc10-83ff-481a-abe5-f0732aef7e05" containerName="revoke-keys" Nov 02 14:15:30.333173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332958 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efc92d0-8b09-4876-95dc-16113a80e34a" containerName="cleanup" Nov 02 14:15:30.333173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.332967 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6cb207-255f-4137-a9a9-88802df4aaa1" containerName="cleanup" Nov 02 14:15:30.336977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.336955 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.359800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.359762 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk"] Nov 02 14:15:30.367803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.367768 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:15:30.367962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.367809 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:15:30.402991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.402970 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.403300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.403012 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz58x\" (UniqueName: \"kubernetes.io/projected/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-kube-api-access-wz58x\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.403300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.403216 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.504125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.504093 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz58x\" (UniqueName: \"kubernetes.io/projected/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-kube-api-access-wz58x\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.504508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.504226 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.504508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.504305 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.504977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.504953 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.506560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.506537 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.543038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.543014 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz58x\" (UniqueName: \"kubernetes.io/projected/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-kube-api-access-wz58x\") pod \"maas-api-revoke-sub-29cc4291-759lk\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.651101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.651078 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:30.968383 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:30.968352 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5961c0c8_a2d3_48e2_95ce_7ae99ec24e23.slice/crio-a035dfd98f2a71105be2b827feddd0e7766775db19418842b5e9898dd97f0127 WatchSource:0}: Error finding container a035dfd98f2a71105be2b827feddd0e7766775db19418842b5e9898dd97f0127: Status 404 returned error can't find the container with id a035dfd98f2a71105be2b827feddd0e7766775db19418842b5e9898dd97f0127 Nov 02 14:15:30.968715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:30.968692 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk"] Nov 02 14:15:31.438636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.438604 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r"] Nov 02 14:15:31.443294 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.443271 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.464370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.464341 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r"] Nov 02 14:15:31.516053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.516018 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.516514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.516092 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.516514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.516253 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72ncs\" (UniqueName: \"kubernetes.io/projected/a162de50-020e-4e2b-8c42-e0145358ca83-kube-api-access-72ncs\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.617477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.617446 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.617639 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.617504 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.617639 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.617565 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72ncs\" (UniqueName: \"kubernetes.io/projected/a162de50-020e-4e2b-8c42-e0145358ca83-kube-api-access-72ncs\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.618139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.618096 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.619824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.619802 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.642491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.642458 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72ncs\" (UniqueName: \"kubernetes.io/projected/a162de50-020e-4e2b-8c42-e0145358ca83-kube-api-access-72ncs\") pod \"maas-api-revoke-sub-9e5428c7-rcv2r\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:31.672047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.671977 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" event={"ID":"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23","Type":"ContainerStarted","Data":"554c0b3df6647cca6e512698fe44fd01f7fd316606b7a2abb3245dc083df2c00"} Nov 02 14:15:31.672047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.672037 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" event={"ID":"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23","Type":"ContainerStarted","Data":"a035dfd98f2a71105be2b827feddd0e7766775db19418842b5e9898dd97f0127"} Nov 02 14:15:31.720794 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.720731 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" podStartSLOduration=1.7207199979999999 podStartE2EDuration="1.720719998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:31.718916898 +0000 UTC m=+1096.756021830" watchObservedRunningTime="2026-11-02 14:15:31.720719998 +0000 UTC m=+1096.757824929" Nov 02 14:15:31.766453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:31.766415 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:32.081430 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:32.081394 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda162de50_020e_4e2b_8c42_e0145358ca83.slice/crio-561f657af9792a18ad71bfc8bcfce21af89f548a161575ac4b3569cc08b95f45 WatchSource:0}: Error finding container 561f657af9792a18ad71bfc8bcfce21af89f548a161575ac4b3569cc08b95f45: Status 404 returned error can't find the container with id 561f657af9792a18ad71bfc8bcfce21af89f548a161575ac4b3569cc08b95f45 Nov 02 14:15:32.081703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:32.081680 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r"] Nov 02 14:15:32.686817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:32.686200 4431 generic.go:356] "Generic (PLEG): container finished" podID="5961c0c8-a2d3-48e2-95ce-7ae99ec24e23" containerID="554c0b3df6647cca6e512698fe44fd01f7fd316606b7a2abb3245dc083df2c00" exitCode=0 Nov 02 14:15:32.686817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:32.686390 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" event={"ID":"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23","Type":"ContainerDied","Data":"554c0b3df6647cca6e512698fe44fd01f7fd316606b7a2abb3245dc083df2c00"} Nov 02 14:15:32.689079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:32.688989 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" event={"ID":"a162de50-020e-4e2b-8c42-e0145358ca83","Type":"ContainerStarted","Data":"faa19b3ef155dcf2a82991dc9ca0907ba034a6e6044baff6e44d5f5266a28c69"} Nov 02 14:15:32.689079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:32.689044 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" event={"ID":"a162de50-020e-4e2b-8c42-e0145358ca83","Type":"ContainerStarted","Data":"561f657af9792a18ad71bfc8bcfce21af89f548a161575ac4b3569cc08b95f45"} Nov 02 14:15:32.785058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:32.785009 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" podStartSLOduration=1.7849943769999999 podStartE2EDuration="1.784994377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:32.779680588 +0000 UTC m=+1097.816785520" watchObservedRunningTime="2026-11-02 14:15:32.784994377 +0000 UTC m=+1097.822099664" Nov 02 14:15:33.697612 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.697578 4431 generic.go:356] "Generic (PLEG): container finished" podID="a162de50-020e-4e2b-8c42-e0145358ca83" containerID="faa19b3ef155dcf2a82991dc9ca0907ba034a6e6044baff6e44d5f5266a28c69" exitCode=0 Nov 02 14:15:33.698018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.697662 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" event={"ID":"a162de50-020e-4e2b-8c42-e0145358ca83","Type":"ContainerDied","Data":"faa19b3ef155dcf2a82991dc9ca0907ba034a6e6044baff6e44d5f5266a28c69"} Nov 02 14:15:33.865285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.865258 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:33.957923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.957848 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-openshift-ca\") pod \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " Nov 02 14:15:33.958078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.957951 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-xks-ca\") pod \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " Nov 02 14:15:33.958078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.958021 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-kube-api-access-wz58x\" (UniqueName: \"kubernetes.io/projected/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-kube-api-access-wz58x\") pod \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\" (UID: \"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23\") " Nov 02 14:15:33.958257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.958236 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-openshift-ca" pod "5961c0c8-a2d3-48e2-95ce-7ae99ec24e23" (UID: "5961c0c8-a2d3-48e2-95ce-7ae99ec24e23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:33.958543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.958523 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:33.960114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.960087 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-xks-ca" pod "5961c0c8-a2d3-48e2-95ce-7ae99ec24e23" (UID: "5961c0c8-a2d3-48e2-95ce-7ae99ec24e23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:15:33.960232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:33.960195 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-kube-api-access-wz58x" pod "5961c0c8-a2d3-48e2-95ce-7ae99ec24e23" (UID: "5961c0c8-a2d3-48e2-95ce-7ae99ec24e23"). InnerVolumeSpecName "kube-api-access-wz58x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:34.059983 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.059961 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:34.059983 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.059983 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz58x\" (UniqueName: \"kubernetes.io/projected/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23-kube-api-access-wz58x\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:34.706503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.706476 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" Nov 02 14:15:34.706941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.706517 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk" event={"ID":"5961c0c8-a2d3-48e2-95ce-7ae99ec24e23","Type":"ContainerDied","Data":"a035dfd98f2a71105be2b827feddd0e7766775db19418842b5e9898dd97f0127"} Nov 02 14:15:34.706941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.706619 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a035dfd98f2a71105be2b827feddd0e7766775db19418842b5e9898dd97f0127" Nov 02 14:15:34.854950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.854922 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:34.971208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.971132 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-openshift-ca\") pod \"a162de50-020e-4e2b-8c42-e0145358ca83\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " Nov 02 14:15:34.971356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.971303 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-xks-ca\") pod \"a162de50-020e-4e2b-8c42-e0145358ca83\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " Nov 02 14:15:34.971427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.971355 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a162de50-020e-4e2b-8c42-e0145358ca83-kube-api-access-72ncs\" (UniqueName: \"kubernetes.io/projected/a162de50-020e-4e2b-8c42-e0145358ca83-kube-api-access-72ncs\") pod \"a162de50-020e-4e2b-8c42-e0145358ca83\" (UID: \"a162de50-020e-4e2b-8c42-e0145358ca83\") " Nov 02 14:15:34.971549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.971522 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-openshift-ca" pod "a162de50-020e-4e2b-8c42-e0145358ca83" (UID: "a162de50-020e-4e2b-8c42-e0145358ca83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:34.971831 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.971812 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:34.973410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.973388 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-xks-ca" pod "a162de50-020e-4e2b-8c42-e0145358ca83" (UID: "a162de50-020e-4e2b-8c42-e0145358ca83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:15:34.973491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:34.973468 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a162de50-020e-4e2b-8c42-e0145358ca83-kube-api-access-72ncs" pod "a162de50-020e-4e2b-8c42-e0145358ca83" (UID: "a162de50-020e-4e2b-8c42-e0145358ca83"). InnerVolumeSpecName "kube-api-access-72ncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:35.073097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:35.073069 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a162de50-020e-4e2b-8c42-e0145358ca83-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:35.073097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:35.073096 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72ncs\" (UniqueName: \"kubernetes.io/projected/a162de50-020e-4e2b-8c42-e0145358ca83-kube-api-access-72ncs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:35.715675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:35.715647 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" Nov 02 14:15:35.716109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:35.715687 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r" event={"ID":"a162de50-020e-4e2b-8c42-e0145358ca83","Type":"ContainerDied","Data":"561f657af9792a18ad71bfc8bcfce21af89f548a161575ac4b3569cc08b95f45"} Nov 02 14:15:35.716109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:35.715735 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561f657af9792a18ad71bfc8bcfce21af89f548a161575ac4b3569cc08b95f45" Nov 02 14:15:38.824318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:38.824290 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:15:44.091421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.091385 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f"] Nov 02 14:15:44.106546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.106522 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f6e542-wbr6f"] Nov 02 14:15:44.980617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.980582 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75"] Nov 02 14:15:44.981173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.981158 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5961c0c8-a2d3-48e2-95ce-7ae99ec24e23" containerName="revoke-keys" Nov 02 14:15:44.981242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.981174 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5961c0c8-a2d3-48e2-95ce-7ae99ec24e23" containerName="revoke-keys" Nov 02 14:15:44.981242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.981199 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a162de50-020e-4e2b-8c42-e0145358ca83" containerName="revoke-keys" Nov 02 14:15:44.981242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.981204 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a162de50-020e-4e2b-8c42-e0145358ca83" containerName="revoke-keys" Nov 02 14:15:44.981348 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.981294 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5961c0c8-a2d3-48e2-95ce-7ae99ec24e23" containerName="revoke-keys" Nov 02 14:15:44.981348 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.981304 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a162de50-020e-4e2b-8c42-e0145358ca83" containerName="revoke-keys" Nov 02 14:15:44.985044 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:44.985026 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.004919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.004896 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75"] Nov 02 14:15:45.013013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.012990 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:15:45.013135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.013117 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:15:45.077327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.077288 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6171bd98-da10-45cf-9b91-7ec7cbe1622d" path="/var/lib/kubelet/pods/6171bd98-da10-45cf-9b91-7ec7cbe1622d/volumes" Nov 02 14:15:45.123654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.123621 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.124020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.123684 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ccr\" (UniqueName: \"kubernetes.io/projected/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-kube-api-access-f2ccr\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.124020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.123797 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.225289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.225256 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ccr\" (UniqueName: \"kubernetes.io/projected/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-kube-api-access-f2ccr\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.225448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.225310 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.225524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.225466 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.226033 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.226010 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.227922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.227900 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.250030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.249982 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ccr\" (UniqueName: \"kubernetes.io/projected/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-kube-api-access-f2ccr\") pod \"maas-api-revoke-sub-72df75c8-x6v75\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.302311 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.302284 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:45.609062 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:45.609027 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d66b80d_f801_49cb_aa1c_9ab9e4fe85ef.slice/crio-ef8caead996f913811869ead154b3f90cbed4764352260c5ae0710df53aaf390 WatchSource:0}: Error finding container ef8caead996f913811869ead154b3f90cbed4764352260c5ae0710df53aaf390: Status 404 returned error can't find the container with id ef8caead996f913811869ead154b3f90cbed4764352260c5ae0710df53aaf390 Nov 02 14:15:45.609656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.609633 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75"] Nov 02 14:15:45.810245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.810217 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm"] Nov 02 14:15:45.816090 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.816065 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" event={"ID":"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef","Type":"ContainerStarted","Data":"ef8caead996f913811869ead154b3f90cbed4764352260c5ae0710df53aaf390"} Nov 02 14:15:45.816211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.816151 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:45.835947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.835923 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm"] Nov 02 14:15:45.933802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.933761 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:45.933923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.933905 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:45.933975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:45.933962 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgncm\" (UniqueName: \"kubernetes.io/projected/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-kube-api-access-hgncm\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:46.034710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.034689 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:46.034832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.034802 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:46.034883 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.034861 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgncm\" (UniqueName: \"kubernetes.io/projected/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-kube-api-access-hgncm\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:46.035525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.035503 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:46.036884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.036866 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:46.062892 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.062855 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgncm\" (UniqueName: \"kubernetes.io/projected/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-kube-api-access-hgncm\") pod \"maas-api-revoke-sub-d8475d54-2w6dm\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:46.128937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.128914 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:46.444271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.444247 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm"] Nov 02 14:15:46.460658 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:46.460632 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc76f74_f1b7_4c92_af67_3e21a2b2a4bf.slice/crio-77594c36dbd938704f166d49af67f50929dc67afdbc74c59d3f2c14e139699af WatchSource:0}: Error finding container 77594c36dbd938704f166d49af67f50929dc67afdbc74c59d3f2c14e139699af: Status 404 returned error can't find the container with id 77594c36dbd938704f166d49af67f50929dc67afdbc74c59d3f2c14e139699af Nov 02 14:15:46.826967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.826934 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" event={"ID":"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf","Type":"ContainerStarted","Data":"9df3b9bb9df87c40fcea8f09b2c255d5667590b1934d238f6abd68a4a1d69bf6"} Nov 02 14:15:46.827073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.826989 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" event={"ID":"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf","Type":"ContainerStarted","Data":"77594c36dbd938704f166d49af67f50929dc67afdbc74c59d3f2c14e139699af"} Nov 02 14:15:46.828916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.828888 4431 generic.go:356] "Generic (PLEG): container finished" podID="0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef" containerID="fb3fc3560139f6024101c00f1f53fb8db4a7f7c8ef52e70b02aaa45fd2b72e75" exitCode=0 Nov 02 14:15:46.829005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.828938 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" event={"ID":"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef","Type":"ContainerDied","Data":"fb3fc3560139f6024101c00f1f53fb8db4a7f7c8ef52e70b02aaa45fd2b72e75"} Nov 02 14:15:46.871724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:46.871678 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" podStartSLOduration=1.8716663 podStartE2EDuration="1.8716663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:46.869483628 +0000 UTC m=+1111.906588573" watchObservedRunningTime="2026-11-02 14:15:46.8716663 +0000 UTC m=+1111.908771251" Nov 02 14:15:47.837588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:47.837554 4431 generic.go:356] "Generic (PLEG): container finished" podID="1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf" containerID="9df3b9bb9df87c40fcea8f09b2c255d5667590b1934d238f6abd68a4a1d69bf6" exitCode=0 Nov 02 14:15:47.838012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:47.837606 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" event={"ID":"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf","Type":"ContainerDied","Data":"9df3b9bb9df87c40fcea8f09b2c255d5667590b1934d238f6abd68a4a1d69bf6"} Nov 02 14:15:47.978656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:47.978630 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:48.159570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.159540 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-xks-ca\") pod \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " Nov 02 14:15:48.159752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.159651 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-kube-api-access-f2ccr\" (UniqueName: \"kubernetes.io/projected/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-kube-api-access-f2ccr\") pod \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " Nov 02 14:15:48.159752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.159679 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-openshift-ca\") pod \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\" (UID: \"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef\") " Nov 02 14:15:48.160111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.160087 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-openshift-ca" pod "0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef" (UID: "0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:48.160272 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.160257 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:48.161596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.161571 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-xks-ca" pod "0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef" (UID: "0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:15:48.161713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.161693 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-kube-api-access-f2ccr" pod "0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef" (UID: "0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef"). InnerVolumeSpecName "kube-api-access-f2ccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:48.261753 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.261729 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:48.261753 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.261753 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2ccr\" (UniqueName: \"kubernetes.io/projected/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef-kube-api-access-f2ccr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:48.845576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.845546 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" Nov 02 14:15:48.845996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.845576 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75" event={"ID":"0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef","Type":"ContainerDied","Data":"ef8caead996f913811869ead154b3f90cbed4764352260c5ae0710df53aaf390"} Nov 02 14:15:48.845996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:48.845623 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef8caead996f913811869ead154b3f90cbed4764352260c5ae0710df53aaf390" Nov 02 14:15:49.009359 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.009338 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:49.173089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.173062 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-openshift-ca\") pod \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " Nov 02 14:15:49.173284 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.173135 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-kube-api-access-hgncm\" (UniqueName: \"kubernetes.io/projected/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-kube-api-access-hgncm\") pod \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " Nov 02 14:15:49.173284 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.173189 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-xks-ca\") pod \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\" (UID: \"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf\") " Nov 02 14:15:49.173484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.173457 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-openshift-ca" pod "1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf" (UID: "1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:15:49.174210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.174015 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:49.175629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.175607 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-kube-api-access-hgncm" pod "1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf" (UID: "1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf"). InnerVolumeSpecName "kube-api-access-hgncm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:15:49.176125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.176105 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-xks-ca" pod "1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf" (UID: "1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:15:49.275613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.275589 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgncm\" (UniqueName: \"kubernetes.io/projected/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-kube-api-access-hgncm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:49.275613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.275611 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:15:49.854667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.854634 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" Nov 02 14:15:49.855079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.854664 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm" event={"ID":"1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf","Type":"ContainerDied","Data":"77594c36dbd938704f166d49af67f50929dc67afdbc74c59d3f2c14e139699af"} Nov 02 14:15:49.855079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:49.854714 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77594c36dbd938704f166d49af67f50929dc67afdbc74c59d3f2c14e139699af" Nov 02 14:15:51.299066 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:15:51.299027 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:15:53.035039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:53.035008 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk"] Nov 02 14:15:53.048019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:53.047991 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b108d0d7-9h6lk"] Nov 02 14:15:53.075183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:53.075155 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06224c7c-dd28-461c-9615-3ed655fab3f6" path="/var/lib/kubelet/pods/06224c7c-dd28-461c-9615-3ed655fab3f6/volumes" Nov 02 14:15:56.385802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.385313 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7"] Nov 02 14:15:56.386266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.386205 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef" containerName="revoke-keys" Nov 02 14:15:56.386266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.386222 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef" containerName="revoke-keys" Nov 02 14:15:56.386266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.386265 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf" containerName="revoke-keys" Nov 02 14:15:56.386452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.386273 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf" containerName="revoke-keys" Nov 02 14:15:56.386452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.386421 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef" containerName="revoke-keys" Nov 02 14:15:56.386452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.386440 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf" containerName="revoke-keys" Nov 02 14:15:56.419060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.419028 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:56.439799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.436704 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7"] Nov 02 14:15:56.439799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.437594 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:15:56.475800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.451015 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:15:56.954411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.951910 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddrrz\" (UniqueName: \"kubernetes.io/projected/04c34465-9e33-4a84-9182-b93afb13b32a-kube-api-access-ddrrz\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:56.954411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.952018 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:56.954411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:56.952069 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:57.120802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.119903 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddrrz\" (UniqueName: \"kubernetes.io/projected/04c34465-9e33-4a84-9182-b93afb13b32a-kube-api-access-ddrrz\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:57.120802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.120052 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:57.120802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.120096 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:57.145800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.122172 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:57.159455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.159425 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddrrz\" (UniqueName: \"kubernetes.io/projected/04c34465-9e33-4a84-9182-b93afb13b32a-kube-api-access-ddrrz\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:57.160944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.160853 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f56c5b-7mjm7\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:57.172150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.172123 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2"] Nov 02 14:15:57.172542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.172525 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-addaa4b1-8k4h2"] Nov 02 14:15:57.265571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.265502 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:15:57.707109 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:15:57.707081 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c34465_9e33_4a84_9182_b93afb13b32a.slice/crio-faac5b16cda339ee44baeeef65c5768e4debeaa508afcf0b70f0b06564bd41d7 WatchSource:0}: Error finding container faac5b16cda339ee44baeeef65c5768e4debeaa508afcf0b70f0b06564bd41d7: Status 404 returned error can't find the container with id faac5b16cda339ee44baeeef65c5768e4debeaa508afcf0b70f0b06564bd41d7 Nov 02 14:15:57.708108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:57.708087 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7"] Nov 02 14:15:58.166540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:58.166504 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" event={"ID":"04c34465-9e33-4a84-9182-b93afb13b32a","Type":"ContainerStarted","Data":"134a2029be470d0465bcd14f7493148058791092c719987985b34028810b207c"} Nov 02 14:15:58.166694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:58.166570 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" event={"ID":"04c34465-9e33-4a84-9182-b93afb13b32a","Type":"ContainerStarted","Data":"faac5b16cda339ee44baeeef65c5768e4debeaa508afcf0b70f0b06564bd41d7"} Nov 02 14:15:58.235704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:58.235619 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" podStartSLOduration=2.235607015 podStartE2EDuration="2.235607015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:58.230851534 +0000 UTC m=+1123.267956463" watchObservedRunningTime="2026-11-02 14:15:58.235607015 +0000 UTC m=+1123.272711950" Nov 02 14:15:59.076663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.076629 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34495bd4-f8d3-4c29-9b14-2d63f112b6a0" path="/var/lib/kubelet/pods/34495bd4-f8d3-4c29-9b14-2d63f112b6a0/volumes" Nov 02 14:15:59.177732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.177696 4431 generic.go:356] "Generic (PLEG): container finished" podID="04c34465-9e33-4a84-9182-b93afb13b32a" containerID="134a2029be470d0465bcd14f7493148058791092c719987985b34028810b207c" exitCode=0 Nov 02 14:15:59.177949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.177812 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" event={"ID":"04c34465-9e33-4a84-9182-b93afb13b32a","Type":"ContainerDied","Data":"134a2029be470d0465bcd14f7493148058791092c719987985b34028810b207c"} Nov 02 14:15:59.937698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.937659 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v"] Nov 02 14:15:59.942424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.942408 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:15:59.970821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.970362 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v"] Nov 02 14:15:59.976005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.975982 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4g8\" (UniqueName: \"kubernetes.io/projected/fd0d6682-a02c-4351-9d73-35f0dd936b60-kube-api-access-6c4g8\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:15:59.976132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.976011 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:15:59.976197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:15:59.976174 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:00.077135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.077104 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:00.077720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.077685 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4g8\" (UniqueName: \"kubernetes.io/projected/fd0d6682-a02c-4351-9d73-35f0dd936b60-kube-api-access-6c4g8\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:00.077866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.077743 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:00.078482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.078459 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:00.079947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.079923 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:00.103437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.103412 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4g8\" (UniqueName: \"kubernetes.io/projected/fd0d6682-a02c-4351-9d73-35f0dd936b60-kube-api-access-6c4g8\") pod \"maas-api-revoke-sub-cff2ad82-k2w8v\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:00.257768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.256517 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:00.353406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.353381 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:16:00.381271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.381243 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-xks-ca\") pod \"04c34465-9e33-4a84-9182-b93afb13b32a\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " Nov 02 14:16:00.381411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.381394 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c34465-9e33-4a84-9182-b93afb13b32a-kube-api-access-ddrrz\" (UniqueName: \"kubernetes.io/projected/04c34465-9e33-4a84-9182-b93afb13b32a-kube-api-access-ddrrz\") pod \"04c34465-9e33-4a84-9182-b93afb13b32a\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " Nov 02 14:16:00.381468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.381437 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-openshift-ca\") pod \"04c34465-9e33-4a84-9182-b93afb13b32a\" (UID: \"04c34465-9e33-4a84-9182-b93afb13b32a\") " Nov 02 14:16:00.382009 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.381888 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-openshift-ca" pod "04c34465-9e33-4a84-9182-b93afb13b32a" (UID: "04c34465-9e33-4a84-9182-b93afb13b32a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:00.383757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.383735 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-xks-ca" pod "04c34465-9e33-4a84-9182-b93afb13b32a" (UID: "04c34465-9e33-4a84-9182-b93afb13b32a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:00.383867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.383800 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c34465-9e33-4a84-9182-b93afb13b32a-kube-api-access-ddrrz" pod "04c34465-9e33-4a84-9182-b93afb13b32a" (UID: "04c34465-9e33-4a84-9182-b93afb13b32a"). InnerVolumeSpecName "kube-api-access-ddrrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:00.483343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.483308 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddrrz\" (UniqueName: \"kubernetes.io/projected/04c34465-9e33-4a84-9182-b93afb13b32a-kube-api-access-ddrrz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:00.483343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.483341 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:00.483556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.483358 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c34465-9e33-4a84-9182-b93afb13b32a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:00.572882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:00.572846 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v"] Nov 02 14:16:00.573015 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:00.572950 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0d6682_a02c_4351_9d73_35f0dd936b60.slice/crio-9883928f544d686b4a7d1dc4163fed0d771a892f039da1f81f71c832c821a272 WatchSource:0}: Error finding container 9883928f544d686b4a7d1dc4163fed0d771a892f039da1f81f71c832c821a272: Status 404 returned error can't find the container with id 9883928f544d686b4a7d1dc4163fed0d771a892f039da1f81f71c832c821a272 Nov 02 14:16:01.195214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:01.195189 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" Nov 02 14:16:01.195564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:01.195222 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7" event={"ID":"04c34465-9e33-4a84-9182-b93afb13b32a","Type":"ContainerDied","Data":"faac5b16cda339ee44baeeef65c5768e4debeaa508afcf0b70f0b06564bd41d7"} Nov 02 14:16:01.195564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:01.195278 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faac5b16cda339ee44baeeef65c5768e4debeaa508afcf0b70f0b06564bd41d7" Nov 02 14:16:01.197300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:01.197238 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" event={"ID":"fd0d6682-a02c-4351-9d73-35f0dd936b60","Type":"ContainerStarted","Data":"23b6589f89bf58bcb2b7331e3ec91d74e79e623e2ebaeb8839e1561fb237fd1a"} Nov 02 14:16:01.197300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:01.197291 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" event={"ID":"fd0d6682-a02c-4351-9d73-35f0dd936b60","Type":"ContainerStarted","Data":"9883928f544d686b4a7d1dc4163fed0d771a892f039da1f81f71c832c821a272"} Nov 02 14:16:01.256450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:01.256403 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" podStartSLOduration=2.256391181 podStartE2EDuration="2.256391181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:01.241138197 +0000 UTC m=+1126.278243129" watchObservedRunningTime="2026-11-02 14:16:01.256391181 +0000 UTC m=+1126.293496150" Nov 02 14:16:02.055595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:02.055559 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:16:02.205989 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:02.205956 4431 generic.go:356] "Generic (PLEG): container finished" podID="fd0d6682-a02c-4351-9d73-35f0dd936b60" containerID="23b6589f89bf58bcb2b7331e3ec91d74e79e623e2ebaeb8839e1561fb237fd1a" exitCode=0 Nov 02 14:16:02.206346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:02.206021 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" event={"ID":"fd0d6682-a02c-4351-9d73-35f0dd936b60","Type":"ContainerDied","Data":"23b6589f89bf58bcb2b7331e3ec91d74e79e623e2ebaeb8839e1561fb237fd1a"} Nov 02 14:16:03.483807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.483757 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:03.548848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.547644 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-xks-ca\") pod \"fd0d6682-a02c-4351-9d73-35f0dd936b60\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " Nov 02 14:16:03.548848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.547763 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-openshift-ca\") pod \"fd0d6682-a02c-4351-9d73-35f0dd936b60\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " Nov 02 14:16:03.548848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.547923 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0d6682-a02c-4351-9d73-35f0dd936b60-kube-api-access-6c4g8\" (UniqueName: \"kubernetes.io/projected/fd0d6682-a02c-4351-9d73-35f0dd936b60-kube-api-access-6c4g8\") pod \"fd0d6682-a02c-4351-9d73-35f0dd936b60\" (UID: \"fd0d6682-a02c-4351-9d73-35f0dd936b60\") " Nov 02 14:16:03.548848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.548465 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-openshift-ca" pod "fd0d6682-a02c-4351-9d73-35f0dd936b60" (UID: "fd0d6682-a02c-4351-9d73-35f0dd936b60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:03.551061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.551014 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0d6682-a02c-4351-9d73-35f0dd936b60-kube-api-access-6c4g8" pod "fd0d6682-a02c-4351-9d73-35f0dd936b60" (UID: "fd0d6682-a02c-4351-9d73-35f0dd936b60"). InnerVolumeSpecName "kube-api-access-6c4g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:03.552504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.552478 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-xks-ca" pod "fd0d6682-a02c-4351-9d73-35f0dd936b60" (UID: "fd0d6682-a02c-4351-9d73-35f0dd936b60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:03.649312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.649284 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:03.649432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.649319 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c4g8\" (UniqueName: \"kubernetes.io/projected/fd0d6682-a02c-4351-9d73-35f0dd936b60-kube-api-access-6c4g8\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:03.649432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:03.649335 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd0d6682-a02c-4351-9d73-35f0dd936b60-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:04.273825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:04.271455 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" event={"ID":"fd0d6682-a02c-4351-9d73-35f0dd936b60","Type":"ContainerDied","Data":"9883928f544d686b4a7d1dc4163fed0d771a892f039da1f81f71c832c821a272"} Nov 02 14:16:04.273825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:04.271519 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9883928f544d686b4a7d1dc4163fed0d771a892f039da1f81f71c832c821a272" Nov 02 14:16:04.273825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:04.271593 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v" Nov 02 14:16:04.299647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:04.299623 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz"] Nov 02 14:16:10.035527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:10.035495 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l"] Nov 02 14:16:10.049695 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:10.049659 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b2b76a9-hs46l"] Nov 02 14:16:11.075394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:11.075363 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924bd9ed-d7d8-439b-a946-416c3296a4e2" path="/var/lib/kubelet/pods/924bd9ed-d7d8-439b-a946-416c3296a4e2/volumes" Nov 02 14:16:13.036298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:13.036267 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn"] Nov 02 14:16:13.048975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:13.048950 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16ce1ca2-8smrn"] Nov 02 14:16:13.075933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:13.075906 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c02f8fc-dc0b-469f-bad2-a0c382126ac1" path="/var/lib/kubelet/pods/5c02f8fc-dc0b-469f-bad2-a0c382126ac1/volumes" Nov 02 14:16:14.980800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:14.979864 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:15.143053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.143010 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Nov 02 14:16:15.171751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.171344 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:15.220064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.220027 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" podUID="5540857d-32be-4620-8c54-872b9a767f3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:15.258381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.220282 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 02 14:16:15.325813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.322351 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:15.793174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.793140 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8"] Nov 02 14:16:15.793724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.793707 4431 cpu_manager.go:414] "containerMap: removing container" podUID="04c34465-9e33-4a84-9182-b93afb13b32a" containerName="revoke-keys" Nov 02 14:16:15.793804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.793725 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c34465-9e33-4a84-9182-b93afb13b32a" containerName="revoke-keys" Nov 02 14:16:15.793804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.793739 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fd0d6682-a02c-4351-9d73-35f0dd936b60" containerName="revoke-keys" Nov 02 14:16:15.793804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.793744 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0d6682-a02c-4351-9d73-35f0dd936b60" containerName="revoke-keys" Nov 02 14:16:15.793904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.793840 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c34465-9e33-4a84-9182-b93afb13b32a" containerName="revoke-keys" Nov 02 14:16:15.793904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.793849 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0d6682-a02c-4351-9d73-35f0dd936b60" containerName="revoke-keys" Nov 02 14:16:15.798746 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.798726 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:15.818356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.818334 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8"] Nov 02 14:16:15.826056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.826029 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:16:15.826158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.826091 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:16:15.933047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.933022 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7d2\" (UniqueName: \"kubernetes.io/projected/22fdcd30-9ebd-404c-a99f-bd9bba80b888-kube-api-access-dx7d2\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:15.933178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.933096 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:15.933220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:15.933200 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:16.034500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:16.034471 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7d2\" (UniqueName: \"kubernetes.io/projected/22fdcd30-9ebd-404c-a99f-bd9bba80b888-kube-api-access-dx7d2\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:16.034884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:16.034548 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:16.034884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:16.034590 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:16.035167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:16.035148 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:16.036881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:16.036863 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:16.059491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:16.059437 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7d2\" (UniqueName: \"kubernetes.io/projected/22fdcd30-9ebd-404c-a99f-bd9bba80b888-kube-api-access-dx7d2\") pod \"maas-api-revoke-sub-11da6b8c-7npc8\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:16.112670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:16.112641 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:16.416334 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:16.416301 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fdcd30_9ebd_404c_a99f_bd9bba80b888.slice/crio-5602c4d9a31e715187abaf49849b6b8b62329e40dd0eed686d3315a7ab53404d WatchSource:0}: Error finding container 5602c4d9a31e715187abaf49849b6b8b62329e40dd0eed686d3315a7ab53404d: Status 404 returned error can't find the container with id 5602c4d9a31e715187abaf49849b6b8b62329e40dd0eed686d3315a7ab53404d Nov 02 14:16:16.417053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:16.417028 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8"] Nov 02 14:16:17.437149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:17.437110 4431 generic.go:356] "Generic (PLEG): container finished" podID="22fdcd30-9ebd-404c-a99f-bd9bba80b888" containerID="0e3028ea63cb01339f6e1b44eb35e3a69ef2411def76bae9cfee874407112282" exitCode=0 Nov 02 14:16:17.437616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:17.437204 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" event={"ID":"22fdcd30-9ebd-404c-a99f-bd9bba80b888","Type":"ContainerDied","Data":"0e3028ea63cb01339f6e1b44eb35e3a69ef2411def76bae9cfee874407112282"} Nov 02 14:16:17.437616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:17.437308 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" event={"ID":"22fdcd30-9ebd-404c-a99f-bd9bba80b888","Type":"ContainerStarted","Data":"5602c4d9a31e715187abaf49849b6b8b62329e40dd0eed686d3315a7ab53404d"} Nov 02 14:16:18.595720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.595696 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:18.666755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.666728 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-xks-ca\") pod \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " Nov 02 14:16:18.666937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.666878 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fdcd30-9ebd-404c-a99f-bd9bba80b888-kube-api-access-dx7d2\" (UniqueName: \"kubernetes.io/projected/22fdcd30-9ebd-404c-a99f-bd9bba80b888-kube-api-access-dx7d2\") pod \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " Nov 02 14:16:18.666937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.666913 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-openshift-ca\") pod \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\" (UID: \"22fdcd30-9ebd-404c-a99f-bd9bba80b888\") " Nov 02 14:16:18.667411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.667338 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-openshift-ca" pod "22fdcd30-9ebd-404c-a99f-bd9bba80b888" (UID: "22fdcd30-9ebd-404c-a99f-bd9bba80b888"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:18.667530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.667471 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:18.669208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.669170 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-xks-ca" pod "22fdcd30-9ebd-404c-a99f-bd9bba80b888" (UID: "22fdcd30-9ebd-404c-a99f-bd9bba80b888"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:18.669333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.669213 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fdcd30-9ebd-404c-a99f-bd9bba80b888-kube-api-access-dx7d2" pod "22fdcd30-9ebd-404c-a99f-bd9bba80b888" (UID: "22fdcd30-9ebd-404c-a99f-bd9bba80b888"). InnerVolumeSpecName "kube-api-access-dx7d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:18.768994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.768929 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fdcd30-9ebd-404c-a99f-bd9bba80b888-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:18.768994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:18.768954 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx7d2\" (UniqueName: \"kubernetes.io/projected/22fdcd30-9ebd-404c-a99f-bd9bba80b888-kube-api-access-dx7d2\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:19.448387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.448354 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" podUID="5540857d-32be-4620-8c54-872b9a767f3e" containerName="main" containerID="cri-o://4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee" gracePeriod=45 Nov 02 14:16:19.453731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.453708 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" Nov 02 14:16:19.453862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.453737 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8" event={"ID":"22fdcd30-9ebd-404c-a99f-bd9bba80b888","Type":"ContainerDied","Data":"5602c4d9a31e715187abaf49849b6b8b62329e40dd0eed686d3315a7ab53404d"} Nov 02 14:16:19.453862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.453813 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5602c4d9a31e715187abaf49849b6b8b62329e40dd0eed686d3315a7ab53404d" Nov 02 14:16:19.717220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.717195 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:16:19.779290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.779259 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-model-cache\") pod \"5540857d-32be-4620-8c54-872b9a767f3e\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " Nov 02 14:16:19.779472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.779323 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-dshm\") pod \"5540857d-32be-4620-8c54-872b9a767f3e\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " Nov 02 14:16:19.779472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.779397 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-tmp-dir\") pod \"5540857d-32be-4620-8c54-872b9a767f3e\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " Nov 02 14:16:19.779472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.779434 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-home\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-home\") pod \"5540857d-32be-4620-8c54-872b9a767f3e\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " Nov 02 14:16:19.779654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.779626 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5540857d-32be-4620-8c54-872b9a767f3e-kube-api-access-xwtwb\" (UniqueName: \"kubernetes.io/projected/5540857d-32be-4620-8c54-872b9a767f3e-kube-api-access-xwtwb\") pod \"5540857d-32be-4620-8c54-872b9a767f3e\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " Nov 02 14:16:19.779710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.779663 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5540857d-32be-4620-8c54-872b9a767f3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5540857d-32be-4620-8c54-872b9a767f3e-tls-certs\") pod \"5540857d-32be-4620-8c54-872b9a767f3e\" (UID: \"5540857d-32be-4620-8c54-872b9a767f3e\") " Nov 02 14:16:19.780242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.780215 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-home" pod "5540857d-32be-4620-8c54-872b9a767f3e" (UID: "5540857d-32be-4620-8c54-872b9a767f3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:19.780627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.780587 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-model-cache" pod "5540857d-32be-4620-8c54-872b9a767f3e" (UID: "5540857d-32be-4620-8c54-872b9a767f3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:19.780764 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.780724 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-tmp-dir" pod "5540857d-32be-4620-8c54-872b9a767f3e" (UID: "5540857d-32be-4620-8c54-872b9a767f3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:19.782923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.782900 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5540857d-32be-4620-8c54-872b9a767f3e-kube-api-access-xwtwb" pod "5540857d-32be-4620-8c54-872b9a767f3e" (UID: "5540857d-32be-4620-8c54-872b9a767f3e"). InnerVolumeSpecName "kube-api-access-xwtwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:19.783189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.783169 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5540857d-32be-4620-8c54-872b9a767f3e-tls-certs" pod "5540857d-32be-4620-8c54-872b9a767f3e" (UID: "5540857d-32be-4620-8c54-872b9a767f3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:19.783544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.783519 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-dshm" pod "5540857d-32be-4620-8c54-872b9a767f3e" (UID: "5540857d-32be-4620-8c54-872b9a767f3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:16:19.881092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.881062 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtwb\" (UniqueName: \"kubernetes.io/projected/5540857d-32be-4620-8c54-872b9a767f3e-kube-api-access-xwtwb\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:19.881092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.881090 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5540857d-32be-4620-8c54-872b9a767f3e-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:19.881266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.881100 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:19.881266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.881108 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:19.881266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.881115 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:19.881266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:19.881123 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5540857d-32be-4620-8c54-872b9a767f3e-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:20.463227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.463193 4431 generic.go:356] "Generic (PLEG): container finished" podID="5540857d-32be-4620-8c54-872b9a767f3e" containerID="4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee" exitCode=0 Nov 02 14:16:20.463425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.463242 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" Nov 02 14:16:20.463425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.463286 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" event={"ID":"5540857d-32be-4620-8c54-872b9a767f3e","Type":"ContainerDied","Data":"4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee"} Nov 02 14:16:20.463425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.463329 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz" event={"ID":"5540857d-32be-4620-8c54-872b9a767f3e","Type":"ContainerDied","Data":"2cd21e069eb582c389175ef8d9fcabec5b2809ed863c0c50b441267f99da12cf"} Nov 02 14:16:20.463425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.463350 4431 scope.go:122] "RemoveContainer" containerID="4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee" Nov 02 14:16:20.478052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.478031 4431 scope.go:122] "RemoveContainer" containerID="4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee" Nov 02 14:16:20.478398 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:16:20.478374 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee\": container with ID starting with 4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee not found: ID does not exist" containerID="4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee" Nov 02 14:16:20.478460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.478410 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee"} err="failed to get container status \"4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee\": rpc error: code = NotFound desc = could not find container \"4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee\": container with ID starting with 4a6d5f40483762a4c1d652da833350d498b4ef31afcabdcaadb27f04bd0ac2ee not found: ID does not exist" Nov 02 14:16:20.522482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.522434 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz"] Nov 02 14:16:20.535322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.535297 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-de8cc7/praxis-route-de8cc7-kserve-84d7894c89-xmwtz"] Nov 02 14:16:20.711547 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.711515 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r"] Nov 02 14:16:20.712211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.712194 4431 cpu_manager.go:414] "containerMap: removing container" podUID="22fdcd30-9ebd-404c-a99f-bd9bba80b888" containerName="revoke-keys" Nov 02 14:16:20.712211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.712212 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fdcd30-9ebd-404c-a99f-bd9bba80b888" containerName="revoke-keys" Nov 02 14:16:20.712316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.712234 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5540857d-32be-4620-8c54-872b9a767f3e" containerName="main" Nov 02 14:16:20.712316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.712240 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5540857d-32be-4620-8c54-872b9a767f3e" containerName="main" Nov 02 14:16:20.712375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.712324 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fdcd30-9ebd-404c-a99f-bd9bba80b888" containerName="revoke-keys" Nov 02 14:16:20.712375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.712335 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5540857d-32be-4620-8c54-872b9a767f3e" containerName="main" Nov 02 14:16:20.716995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.716948 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.733096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.733070 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:16:20.733578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.733222 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:16:20.738312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.738292 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r"] Nov 02 14:16:20.793541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.793509 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.793708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.793553 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.793753 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.793729 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swq9v\" (UniqueName: \"kubernetes.io/projected/5240b296-8225-47de-b5b9-18a282d82a95-kube-api-access-swq9v\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.895340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.895307 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.895507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.895348 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.895507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.895432 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swq9v\" (UniqueName: \"kubernetes.io/projected/5240b296-8225-47de-b5b9-18a282d82a95-kube-api-access-swq9v\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.896103 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.896078 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.897675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.897656 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:20.920228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:20.920202 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swq9v\" (UniqueName: \"kubernetes.io/projected/5240b296-8225-47de-b5b9-18a282d82a95-kube-api-access-swq9v\") pod \"maas-api-revoke-sub-ff2806df-2vr2r\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:21.030004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:21.029936 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:21.076317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:21.076286 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5540857d-32be-4620-8c54-872b9a767f3e" path="/var/lib/kubelet/pods/5540857d-32be-4620-8c54-872b9a767f3e/volumes" Nov 02 14:16:21.282818 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:16:21.282718 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:16:21.333586 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:21.333556 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5240b296_8225_47de_b5b9_18a282d82a95.slice/crio-e4917e781267bd49d4cf8bd98479423f914a25490ee3ba16fff10e6a9d826472 WatchSource:0}: Error finding container e4917e781267bd49d4cf8bd98479423f914a25490ee3ba16fff10e6a9d826472: Status 404 returned error can't find the container with id e4917e781267bd49d4cf8bd98479423f914a25490ee3ba16fff10e6a9d826472 Nov 02 14:16:21.334292 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:21.334272 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r"] Nov 02 14:16:21.473178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:21.473143 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" event={"ID":"5240b296-8225-47de-b5b9-18a282d82a95","Type":"ContainerStarted","Data":"e4917e781267bd49d4cf8bd98479423f914a25490ee3ba16fff10e6a9d826472"} Nov 02 14:16:22.481599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:22.481520 4431 generic.go:356] "Generic (PLEG): container finished" podID="5240b296-8225-47de-b5b9-18a282d82a95" containerID="40540e9241d678f8f4b0a12e1dffb71404dc8d5a797b8b90111225bbada4eaff" exitCode=0 Nov 02 14:16:22.481992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:22.481609 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" event={"ID":"5240b296-8225-47de-b5b9-18a282d82a95","Type":"ContainerDied","Data":"40540e9241d678f8f4b0a12e1dffb71404dc8d5a797b8b90111225bbada4eaff"} Nov 02 14:16:23.034262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.034237 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx"] Nov 02 14:16:23.047276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.047250 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08845dc3-kj8mx"] Nov 02 14:16:23.076950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.076924 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd91cb61-2547-4361-bb8e-1bebae482cac" path="/var/lib/kubelet/pods/fd91cb61-2547-4361-bb8e-1bebae482cac/volumes" Nov 02 14:16:23.669504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.669484 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:23.829057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.828986 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-openshift-ca\") pod \"5240b296-8225-47de-b5b9-18a282d82a95\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " Nov 02 14:16:23.829192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.829062 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-xks-ca\") pod \"5240b296-8225-47de-b5b9-18a282d82a95\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " Nov 02 14:16:23.829232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.829219 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5240b296-8225-47de-b5b9-18a282d82a95-kube-api-access-swq9v\" (UniqueName: \"kubernetes.io/projected/5240b296-8225-47de-b5b9-18a282d82a95-kube-api-access-swq9v\") pod \"5240b296-8225-47de-b5b9-18a282d82a95\" (UID: \"5240b296-8225-47de-b5b9-18a282d82a95\") " Nov 02 14:16:23.829380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.829359 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-openshift-ca" pod "5240b296-8225-47de-b5b9-18a282d82a95" (UID: "5240b296-8225-47de-b5b9-18a282d82a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:23.829601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.829585 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:23.830963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.830939 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-xks-ca" pod "5240b296-8225-47de-b5b9-18a282d82a95" (UID: "5240b296-8225-47de-b5b9-18a282d82a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:23.831152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.831130 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5240b296-8225-47de-b5b9-18a282d82a95-kube-api-access-swq9v" pod "5240b296-8225-47de-b5b9-18a282d82a95" (UID: "5240b296-8225-47de-b5b9-18a282d82a95"). InnerVolumeSpecName "kube-api-access-swq9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:23.930368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.930342 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5240b296-8225-47de-b5b9-18a282d82a95-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:23.930368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:23.930367 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swq9v\" (UniqueName: \"kubernetes.io/projected/5240b296-8225-47de-b5b9-18a282d82a95-kube-api-access-swq9v\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:24.512146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:24.512112 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" Nov 02 14:16:24.512331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:24.512155 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r" event={"ID":"5240b296-8225-47de-b5b9-18a282d82a95","Type":"ContainerDied","Data":"e4917e781267bd49d4cf8bd98479423f914a25490ee3ba16fff10e6a9d826472"} Nov 02 14:16:24.512331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:24.512234 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4917e781267bd49d4cf8bd98479423f914a25490ee3ba16fff10e6a9d826472" Nov 02 14:16:26.171030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.170996 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt"] Nov 02 14:16:26.171636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.171613 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5240b296-8225-47de-b5b9-18a282d82a95" containerName="revoke-keys" Nov 02 14:16:26.171793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.171724 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5240b296-8225-47de-b5b9-18a282d82a95" containerName="revoke-keys" Nov 02 14:16:26.171955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.171933 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5240b296-8225-47de-b5b9-18a282d82a95" containerName="revoke-keys" Nov 02 14:16:26.176105 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.176084 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.192464 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.192444 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-866159-openshift-default-dockercfg-jpxc8\"" Nov 02 14:16:26.212448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.212422 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt"] Nov 02 14:16:26.355807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.355764 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.355964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.355809 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffmp\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-kube-api-access-cffmp\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.355964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.355829 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.356039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.356009 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.356073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.356049 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.356073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.356068 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.356187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.356173 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.356238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.356221 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.356282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.356269 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.457500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457436 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.457500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457475 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.457500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457493 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.457733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457533 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.457733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457568 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.457733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457585 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.457733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457611 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.457733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457634 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffmp\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-kube-api-access-cffmp\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.458008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.457771 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.458148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.458128 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.458234 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.458212 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.458285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.458259 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.458343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.458281 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.458401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.458382 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.460124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.460102 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.460215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.460192 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.486034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.486013 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.487484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.487462 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffmp\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-kube-api-access-cffmp\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.493471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.493453 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:26.817830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.817803 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt"] Nov 02 14:16:26.817956 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:26.817926 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f32b447_e76e_42bc_a8fb_61a7ba9e9898.slice/crio-1e6a92df939379866ef29fd980f5adae62a0c29bd463adfdf7bda067ff2f9f16 WatchSource:0}: Error finding container 1e6a92df939379866ef29fd980f5adae62a0c29bd463adfdf7bda067ff2f9f16: Status 404 returned error can't find the container with id 1e6a92df939379866ef29fd980f5adae62a0c29bd463adfdf7bda067ff2f9f16 Nov 02 14:16:26.820236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.820205 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:16:26.820306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.820270 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:16:26.820339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:26.820320 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:16:27.545962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:27.545927 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" event={"ID":"2f32b447-e76e-42bc-a8fb-61a7ba9e9898","Type":"ContainerStarted","Data":"fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243"} Nov 02 14:16:27.546335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:27.546015 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" event={"ID":"2f32b447-e76e-42bc-a8fb-61a7ba9e9898","Type":"ContainerStarted","Data":"1e6a92df939379866ef29fd980f5adae62a0c29bd463adfdf7bda067ff2f9f16"} Nov 02 14:16:27.613023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:27.612952 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podStartSLOduration=1.612932692 podStartE2EDuration="1.612932692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:27.610084996 +0000 UTC m=+1152.647189928" watchObservedRunningTime="2026-11-02 14:16:27.612932692 +0000 UTC m=+1152.650037633" Nov 02 14:16:28.495076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:28.495040 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:28.496323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:28.496289 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:16:28.496480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:28.496350 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 14:16:28.606408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:28.606379 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:16:29.500826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:29.496134 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:16:29.500826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:29.496193 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 14:16:30.498801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:30.498382 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:16:30.498801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:30.498441 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 14:16:31.494098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.494062 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:16:31.494378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.494346 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 14:16:31.584937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.584901 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:16:31.585618 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.585587 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:31.590154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.590128 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f"] Nov 02 14:16:31.605244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.605217 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.002919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.765002 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:16:32.003145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.765131 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:16:32.003657 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.887954 4431 patch_prober.go:29] interesting pod/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:16:32.013916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.013876 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.053826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:31.759769 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f"] Nov 02 14:16:32.053826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.029068 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.172983 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.172928 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.199811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.194930 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:16:32.199811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.195012 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.235802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.232101 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.294809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.264884 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:16:32.295070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.295029 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.477915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.457628 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqjk\" (UniqueName: \"kubernetes.io/projected/19042772-9e2a-4685-b598-6bef25619c28-kube-api-access-vzqjk\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.487129 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.487092 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19042772-9e2a-4685-b598-6bef25619c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.487380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.487361 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19042772-9e2a-4685-b598-6bef25619c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.516627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.512799 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:16:32.516627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.512873 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 14:16:32.516627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.513262 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 14:16:32.529806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.521861 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:16:32.619427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.619382 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19042772-9e2a-4685-b598-6bef25619c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.650168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.630122 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqjk\" (UniqueName: \"kubernetes.io/projected/19042772-9e2a-4685-b598-6bef25619c28-kube-api-access-vzqjk\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.650168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.630185 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19042772-9e2a-4685-b598-6bef25619c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.650168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.631025 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19042772-9e2a-4685-b598-6bef25619c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.667257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.667194 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqjk\" (UniqueName: \"kubernetes.io/projected/19042772-9e2a-4685-b598-6bef25619c28-kube-api-access-vzqjk\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.720801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.718703 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19042772-9e2a-4685-b598-6bef25619c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7135e531-4r78f\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:32.730839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:32.729915 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:33.105731 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:33.105699 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19042772_9e2a_4685_b598_6bef25619c28.slice/crio-554e7276c91b0947e357ba84c5eb32e4af0f0a508cd6b02df273f4d40abc1873 WatchSource:0}: Error finding container 554e7276c91b0947e357ba84c5eb32e4af0f0a508cd6b02df273f4d40abc1873: Status 404 returned error can't find the container with id 554e7276c91b0947e357ba84c5eb32e4af0f0a508cd6b02df273f4d40abc1873 Nov 02 14:16:33.107456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:33.107429 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f"] Nov 02 14:16:33.495223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:33.495192 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:16:33.495515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:33.495469 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 14:16:34.033951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:34.033918 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l"] Nov 02 14:16:34.046367 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:34.046345 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44150d36-pzv9l"] Nov 02 14:16:34.117463 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:34.117435 4431 generic.go:356] "Generic (PLEG): container finished" podID="19042772-9e2a-4685-b598-6bef25619c28" containerID="12c42f377e5c75dc191fd547186ced9c34010e84956fcceb0966ab8a7e51e1ad" exitCode=0 Nov 02 14:16:34.117665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:34.117520 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" event={"ID":"19042772-9e2a-4685-b598-6bef25619c28","Type":"ContainerDied","Data":"12c42f377e5c75dc191fd547186ced9c34010e84956fcceb0966ab8a7e51e1ad"} Nov 02 14:16:34.117665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:34.117584 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" event={"ID":"19042772-9e2a-4685-b598-6bef25619c28","Type":"ContainerStarted","Data":"554e7276c91b0947e357ba84c5eb32e4af0f0a508cd6b02df273f4d40abc1873"} Nov 02 14:16:34.494256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:34.494224 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 02 14:16:34.494502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:34.494279 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 02 14:16:35.076137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.076105 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e239d438-e725-48ee-93f0-a41c6163722a" path="/var/lib/kubelet/pods/e239d438-e725-48ee-93f0-a41c6163722a/volumes" Nov 02 14:16:35.278789 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.278759 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:35.363654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.363590 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19042772-9e2a-4685-b598-6bef25619c28-kube-api-access-vzqjk\" (UniqueName: \"kubernetes.io/projected/19042772-9e2a-4685-b598-6bef25619c28-kube-api-access-vzqjk\") pod \"19042772-9e2a-4685-b598-6bef25619c28\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " Nov 02 14:16:35.363796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.363690 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19042772-9e2a-4685-b598-6bef25619c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19042772-9e2a-4685-b598-6bef25619c28-maas-api-xks-ca\") pod \"19042772-9e2a-4685-b598-6bef25619c28\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " Nov 02 14:16:35.363868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.363855 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19042772-9e2a-4685-b598-6bef25619c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19042772-9e2a-4685-b598-6bef25619c28-maas-api-openshift-ca\") pod \"19042772-9e2a-4685-b598-6bef25619c28\" (UID: \"19042772-9e2a-4685-b598-6bef25619c28\") " Nov 02 14:16:35.364255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.364222 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19042772-9e2a-4685-b598-6bef25619c28-maas-api-openshift-ca" pod "19042772-9e2a-4685-b598-6bef25619c28" (UID: "19042772-9e2a-4685-b598-6bef25619c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:35.364387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.364318 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19042772-9e2a-4685-b598-6bef25619c28-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:35.365706 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.365678 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19042772-9e2a-4685-b598-6bef25619c28-maas-api-xks-ca" pod "19042772-9e2a-4685-b598-6bef25619c28" (UID: "19042772-9e2a-4685-b598-6bef25619c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:35.365849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.365713 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19042772-9e2a-4685-b598-6bef25619c28-kube-api-access-vzqjk" pod "19042772-9e2a-4685-b598-6bef25619c28" (UID: "19042772-9e2a-4685-b598-6bef25619c28"). InnerVolumeSpecName "kube-api-access-vzqjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:35.465846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.465813 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzqjk\" (UniqueName: \"kubernetes.io/projected/19042772-9e2a-4685-b598-6bef25619c28-kube-api-access-vzqjk\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:35.465846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.465842 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19042772-9e2a-4685-b598-6bef25619c28-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:35.499813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.499768 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:35.500288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.500268 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:35.500866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:35.500847 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:16:36.040701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:36.040667 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk"] Nov 02 14:16:36.049951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:36.049920 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eed019-bpvnk"] Nov 02 14:16:36.133717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:36.133693 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" Nov 02 14:16:36.134091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:36.133730 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f" event={"ID":"19042772-9e2a-4685-b598-6bef25619c28","Type":"ContainerDied","Data":"554e7276c91b0947e357ba84c5eb32e4af0f0a508cd6b02df273f4d40abc1873"} Nov 02 14:16:36.134091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:36.133786 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="554e7276c91b0947e357ba84c5eb32e4af0f0a508cd6b02df273f4d40abc1873" Nov 02 14:16:37.075953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:37.075921 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f595db-ea5d-4d04-a03e-b05a65dd5ce5" path="/var/lib/kubelet/pods/47f595db-ea5d-4d04-a03e-b05a65dd5ce5/volumes" Nov 02 14:16:39.379058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.379016 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh"] Nov 02 14:16:39.380449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.380418 4431 cpu_manager.go:414] "containerMap: removing container" podUID="19042772-9e2a-4685-b598-6bef25619c28" containerName="revoke-keys" Nov 02 14:16:39.380609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.380594 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19042772-9e2a-4685-b598-6bef25619c28" containerName="revoke-keys" Nov 02 14:16:39.380958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.380923 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="19042772-9e2a-4685-b598-6bef25619c28" containerName="revoke-keys" Nov 02 14:16:39.385647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.385628 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.407344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.407311 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"openshift-service-ca.crt\"" Nov 02 14:16:39.407891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.407866 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"kube-root-ca.crt\"" Nov 02 14:16:39.412122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.412097 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh"] Nov 02 14:16:39.440487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.440461 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"default-dockercfg-9x58g\"" Nov 02 14:16:39.450153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.450115 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"distinct-w3-866159-kserve-self-signed-certs\"" Nov 02 14:16:39.517943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.517918 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-dshm\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.518101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.518009 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tls-certs\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.518101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.518050 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tmp-dir\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.518101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.518080 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2b2s\" (UniqueName: \"kubernetes.io/projected/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-kube-api-access-t2b2s\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.518273 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.518202 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-model-cache\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.518273 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.518255 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-home\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.619365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.619316 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-model-cache\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.619574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.619436 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-home\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.619574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.619514 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-dshm\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.619709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.619608 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tls-certs\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.619709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.619645 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tmp-dir\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.619709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.619698 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2b2s\" (UniqueName: \"kubernetes.io/projected/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-kube-api-access-t2b2s\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.621289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.620540 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-model-cache\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.621289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.620810 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-home\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.621959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.621935 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tmp-dir\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.624206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.624181 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-dshm\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.624486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.624449 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tls-certs\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.644878 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.644853 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2b2s\" (UniqueName: \"kubernetes.io/projected/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-kube-api-access-t2b2s\") pod \"distinct-w3-866159-kserve-5857f4f7f-bxvjh\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:39.710911 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:39.707764 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:40.036212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.036184 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh"] Nov 02 14:16:40.037418 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:40.037392 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd37e3fc7_7ac2_4b12_90a6_78e8730dc69c.slice/crio-af1c9a3d02040d3c271b281dd09600141a533a9a6f1e67738b5a4a7a0b860996 WatchSource:0}: Error finding container af1c9a3d02040d3c271b281dd09600141a533a9a6f1e67738b5a4a7a0b860996: Status 404 returned error can't find the container with id af1c9a3d02040d3c271b281dd09600141a533a9a6f1e67738b5a4a7a0b860996 Nov 02 14:16:40.169135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.168944 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" event={"ID":"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c","Type":"ContainerStarted","Data":"f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55"} Nov 02 14:16:40.169135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.168990 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" event={"ID":"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c","Type":"ContainerStarted","Data":"af1c9a3d02040d3c271b281dd09600141a533a9a6f1e67738b5a4a7a0b860996"} Nov 02 14:16:40.709921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.709889 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:40.818102 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.818056 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podStartSLOduration=1.818042027 podStartE2EDuration="1.818042027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:40.219949149 +0000 UTC m=+1165.257054081" watchObservedRunningTime="2026-11-02 14:16:40.818042027 +0000 UTC m=+1165.855146959" Nov 02 14:16:40.821324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.821302 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm"] Nov 02 14:16:40.825596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.825579 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:40.844416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.844388 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:16:40.844533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.844520 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:16:40.848440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.848419 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm"] Nov 02 14:16:40.935614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.935576 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:40.935845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.935725 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5n5\" (UniqueName: \"kubernetes.io/projected/3092e93f-2bef-4d31-b63b-355e989d5b18-kube-api-access-fd5n5\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:40.935845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:40.935763 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:41.037617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:41.037539 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:41.037617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:41.037592 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5n5\" (UniqueName: \"kubernetes.io/projected/3092e93f-2bef-4d31-b63b-355e989d5b18-kube-api-access-fd5n5\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:41.037617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:41.037618 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:41.038300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:41.038273 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:41.039999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:41.039976 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:41.062619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:41.062588 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5n5\" (UniqueName: \"kubernetes.io/projected/3092e93f-2bef-4d31-b63b-355e989d5b18-kube-api-access-fd5n5\") pod \"maas-api-revoke-sub-877c2007-j2fgm\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:41.139638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:41.139614 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:41.445539 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:41.445510 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3092e93f_2bef_4d31_b63b_355e989d5b18.slice/crio-294d98e51c084bce8c8f42867e8130423d12e1834df720be961aaaf6ab91a843 WatchSource:0}: Error finding container 294d98e51c084bce8c8f42867e8130423d12e1834df720be961aaaf6ab91a843: Status 404 returned error can't find the container with id 294d98e51c084bce8c8f42867e8130423d12e1834df720be961aaaf6ab91a843 Nov 02 14:16:41.445824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:41.445803 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm"] Nov 02 14:16:42.186472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:42.186438 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" event={"ID":"3092e93f-2bef-4d31-b63b-355e989d5b18","Type":"ContainerStarted","Data":"c3a7dfd38db0c573ebd0eb9ef9efc297afbe0f83d3764bb47536e522e6e67144"} Nov 02 14:16:42.186895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:42.186497 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" event={"ID":"3092e93f-2bef-4d31-b63b-355e989d5b18","Type":"ContainerStarted","Data":"294d98e51c084bce8c8f42867e8130423d12e1834df720be961aaaf6ab91a843"} Nov 02 14:16:42.234117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:42.234069 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" podStartSLOduration=2.23405472 podStartE2EDuration="2.23405472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:42.228669189 +0000 UTC m=+1167.265774132" watchObservedRunningTime="2026-11-02 14:16:42.23405472 +0000 UTC m=+1167.271159651" Nov 02 14:16:43.195189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:43.195154 4431 generic.go:356] "Generic (PLEG): container finished" podID="3092e93f-2bef-4d31-b63b-355e989d5b18" containerID="c3a7dfd38db0c573ebd0eb9ef9efc297afbe0f83d3764bb47536e522e6e67144" exitCode=0 Nov 02 14:16:43.195606 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:43.195238 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" event={"ID":"3092e93f-2bef-4d31-b63b-355e989d5b18","Type":"ContainerDied","Data":"c3a7dfd38db0c573ebd0eb9ef9efc297afbe0f83d3764bb47536e522e6e67144"} Nov 02 14:16:44.388031 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.388002 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:44.490220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.490144 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3092e93f-2bef-4d31-b63b-355e989d5b18-kube-api-access-fd5n5\" (UniqueName: \"kubernetes.io/projected/3092e93f-2bef-4d31-b63b-355e989d5b18-kube-api-access-fd5n5\") pod \"3092e93f-2bef-4d31-b63b-355e989d5b18\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " Nov 02 14:16:44.490372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.490247 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-openshift-ca\") pod \"3092e93f-2bef-4d31-b63b-355e989d5b18\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " Nov 02 14:16:44.490372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.490355 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-xks-ca\") pod \"3092e93f-2bef-4d31-b63b-355e989d5b18\" (UID: \"3092e93f-2bef-4d31-b63b-355e989d5b18\") " Nov 02 14:16:44.490616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.490590 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-openshift-ca" pod "3092e93f-2bef-4d31-b63b-355e989d5b18" (UID: "3092e93f-2bef-4d31-b63b-355e989d5b18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:44.490872 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.490851 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:44.492412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.492385 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-xks-ca" pod "3092e93f-2bef-4d31-b63b-355e989d5b18" (UID: "3092e93f-2bef-4d31-b63b-355e989d5b18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:44.492510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.492472 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3092e93f-2bef-4d31-b63b-355e989d5b18-kube-api-access-fd5n5" pod "3092e93f-2bef-4d31-b63b-355e989d5b18" (UID: "3092e93f-2bef-4d31-b63b-355e989d5b18"). InnerVolumeSpecName "kube-api-access-fd5n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:44.592255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.592230 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd5n5\" (UniqueName: \"kubernetes.io/projected/3092e93f-2bef-4d31-b63b-355e989d5b18-kube-api-access-fd5n5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:44.592255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:44.592255 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3092e93f-2bef-4d31-b63b-355e989d5b18-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:45.211732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:45.211704 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" Nov 02 14:16:45.211912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:45.211739 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm" event={"ID":"3092e93f-2bef-4d31-b63b-355e989d5b18","Type":"ContainerDied","Data":"294d98e51c084bce8c8f42867e8130423d12e1834df720be961aaaf6ab91a843"} Nov 02 14:16:45.211912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:45.211804 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="294d98e51c084bce8c8f42867e8130423d12e1834df720be961aaaf6ab91a843" Nov 02 14:16:46.033314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.033283 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv"] Nov 02 14:16:46.046181 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.046153 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5b8af3b-v77vv"] Nov 02 14:16:46.405082 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.405050 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq"] Nov 02 14:16:46.405619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.405603 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3092e93f-2bef-4d31-b63b-355e989d5b18" containerName="revoke-keys" Nov 02 14:16:46.405669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.405620 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3092e93f-2bef-4d31-b63b-355e989d5b18" containerName="revoke-keys" Nov 02 14:16:46.405729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.405720 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3092e93f-2bef-4d31-b63b-355e989d5b18" containerName="revoke-keys" Nov 02 14:16:46.409332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.409315 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.432046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.432020 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:16:46.432259 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.432214 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:16:46.440379 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.440354 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq"] Nov 02 14:16:46.519648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.519622 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzjbm\" (UniqueName: \"kubernetes.io/projected/cd209d2f-9ec4-41e5-9135-cc0752e384c6-kube-api-access-vzjbm\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.519764 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.519749 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.519922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.519902 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.621295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.621265 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.621423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.621339 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.621423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.621416 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzjbm\" (UniqueName: \"kubernetes.io/projected/cd209d2f-9ec4-41e5-9135-cc0752e384c6-kube-api-access-vzjbm\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.622064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.622036 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.623562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.623536 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.646053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.646030 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzjbm\" (UniqueName: \"kubernetes.io/projected/cd209d2f-9ec4-41e5-9135-cc0752e384c6-kube-api-access-vzjbm\") pod \"maas-api-revoke-sub-62d72354-5f2dq\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:46.722633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:46.722571 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:47.031772 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:47.031734 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd209d2f_9ec4_41e5_9135_cc0752e384c6.slice/crio-ae385fd14b4d3a1a263d2ebb2879501e56e489f2ccc975df94105b2b2e5b4592 WatchSource:0}: Error finding container ae385fd14b4d3a1a263d2ebb2879501e56e489f2ccc975df94105b2b2e5b4592: Status 404 returned error can't find the container with id ae385fd14b4d3a1a263d2ebb2879501e56e489f2ccc975df94105b2b2e5b4592 Nov 02 14:16:47.031970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:47.031944 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq"] Nov 02 14:16:47.081651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:47.081626 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d80f20f-9f84-465d-8fb6-f93ac92b2140" path="/var/lib/kubelet/pods/5d80f20f-9f84-465d-8fb6-f93ac92b2140/volumes" Nov 02 14:16:47.231587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:47.231553 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" event={"ID":"cd209d2f-9ec4-41e5-9135-cc0752e384c6","Type":"ContainerStarted","Data":"ae385fd14b4d3a1a263d2ebb2879501e56e489f2ccc975df94105b2b2e5b4592"} Nov 02 14:16:48.240588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:48.240511 4431 generic.go:356] "Generic (PLEG): container finished" podID="cd209d2f-9ec4-41e5-9135-cc0752e384c6" containerID="673bc71c2a70e7b428c092d151cd20ef40d1204c142e95e46bddec021c7fe066" exitCode=0 Nov 02 14:16:48.240960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:48.240586 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" event={"ID":"cd209d2f-9ec4-41e5-9135-cc0752e384c6","Type":"ContainerDied","Data":"673bc71c2a70e7b428c092d151cd20ef40d1204c142e95e46bddec021c7fe066"} Nov 02 14:16:49.042754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:49.040374 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk"] Nov 02 14:16:49.051915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:49.051886 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3b20bc0-788vk"] Nov 02 14:16:49.085918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:49.085879 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53392b50-8434-40be-b5f8-cbfce8e003cd" path="/var/lib/kubelet/pods/53392b50-8434-40be-b5f8-cbfce8e003cd/volumes" Nov 02 14:16:49.739732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:49.738929 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:49.915728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:49.915698 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:50.112806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.110717 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn"] Nov 02 14:16:50.135805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.134435 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:50.168851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.166235 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f56f5f-s5ctn"] Nov 02 14:16:50.278645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.278618 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" Nov 02 14:16:50.279358 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.279334 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq" event={"ID":"cd209d2f-9ec4-41e5-9135-cc0752e384c6","Type":"ContainerDied","Data":"ae385fd14b4d3a1a263d2ebb2879501e56e489f2ccc975df94105b2b2e5b4592"} Nov 02 14:16:50.279557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.279542 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae385fd14b4d3a1a263d2ebb2879501e56e489f2ccc975df94105b2b2e5b4592" Nov 02 14:16:50.298514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.298482 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-openshift-ca\") pod \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " Nov 02 14:16:50.298745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.298725 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-xks-ca\") pod \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " Nov 02 14:16:50.298968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.298947 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd209d2f-9ec4-41e5-9135-cc0752e384c6-kube-api-access-vzjbm\" (UniqueName: \"kubernetes.io/projected/cd209d2f-9ec4-41e5-9135-cc0752e384c6-kube-api-access-vzjbm\") pod \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\" (UID: \"cd209d2f-9ec4-41e5-9135-cc0752e384c6\") " Nov 02 14:16:50.300516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.300493 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-openshift-ca" pod "cd209d2f-9ec4-41e5-9135-cc0752e384c6" (UID: "cd209d2f-9ec4-41e5-9135-cc0752e384c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:50.309751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.307711 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-xks-ca" pod "cd209d2f-9ec4-41e5-9135-cc0752e384c6" (UID: "cd209d2f-9ec4-41e5-9135-cc0752e384c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:50.316772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.316747 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd209d2f-9ec4-41e5-9135-cc0752e384c6-kube-api-access-vzjbm" pod "cd209d2f-9ec4-41e5-9135-cc0752e384c6" (UID: "cd209d2f-9ec4-41e5-9135-cc0752e384c6"). InnerVolumeSpecName "kube-api-access-vzjbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:50.319955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.319532 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:16:50.400871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.400841 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:50.400871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.400872 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd209d2f-9ec4-41e5-9135-cc0752e384c6-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:50.401070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:50.400886 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzjbm\" (UniqueName: \"kubernetes.io/projected/cd209d2f-9ec4-41e5-9135-cc0752e384c6-kube-api-access-vzjbm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:51.076023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:51.075990 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625ce4aa-7e7e-444e-91ca-767367e2ca01" path="/var/lib/kubelet/pods/625ce4aa-7e7e-444e-91ca-767367e2ca01/volumes" Nov 02 14:16:51.284342 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:16:51.284296 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:16:56.797215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.797185 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj"] Nov 02 14:16:56.797746 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.797732 4431 cpu_manager.go:414] "containerMap: removing container" podUID="cd209d2f-9ec4-41e5-9135-cc0752e384c6" containerName="revoke-keys" Nov 02 14:16:56.797815 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.797747 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd209d2f-9ec4-41e5-9135-cc0752e384c6" containerName="revoke-keys" Nov 02 14:16:56.797908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.797898 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd209d2f-9ec4-41e5-9135-cc0752e384c6" containerName="revoke-keys" Nov 02 14:16:56.802687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.802669 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:56.823280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.823256 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:16:56.836233 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.836203 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:16:56.840926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.840903 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj"] Nov 02 14:16:56.868508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.868488 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnrd\" (UniqueName: \"kubernetes.io/projected/098df760-9588-4719-9283-6dec0c8676e4-kube-api-access-2lnrd\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:56.868684 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.868660 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098df760-9588-4719-9283-6dec0c8676e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:56.868867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.868847 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098df760-9588-4719-9283-6dec0c8676e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:56.970269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.970244 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnrd\" (UniqueName: \"kubernetes.io/projected/098df760-9588-4719-9283-6dec0c8676e4-kube-api-access-2lnrd\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:56.970361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.970329 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098df760-9588-4719-9283-6dec0c8676e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:56.970450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.970434 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098df760-9588-4719-9283-6dec0c8676e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:56.971260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.971238 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098df760-9588-4719-9283-6dec0c8676e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:56.973497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:56.973472 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098df760-9588-4719-9283-6dec0c8676e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:57.004910 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:57.004876 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnrd\" (UniqueName: \"kubernetes.io/projected/098df760-9588-4719-9283-6dec0c8676e4-kube-api-access-2lnrd\") pod \"maas-api-revoke-sub-c5fef3f9-cgjqj\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:57.116196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:57.116140 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:57.431323 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:16:57.431295 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098df760_9588_4719_9283_6dec0c8676e4.slice/crio-915b48be3961a3f7417a23ab0996c4228ff985c40eb4fb943bb17c21f46fd377 WatchSource:0}: Error finding container 915b48be3961a3f7417a23ab0996c4228ff985c40eb4fb943bb17c21f46fd377: Status 404 returned error can't find the container with id 915b48be3961a3f7417a23ab0996c4228ff985c40eb4fb943bb17c21f46fd377 Nov 02 14:16:57.433981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:57.433953 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj"] Nov 02 14:16:58.345756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:58.345730 4431 generic.go:356] "Generic (PLEG): container finished" podID="098df760-9588-4719-9283-6dec0c8676e4" containerID="8128b561fa1fabfd8a710ffc744153e07dc5ad8c5ece71556d3079a2090c3a30" exitCode=0 Nov 02 14:16:58.346092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:58.345819 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" event={"ID":"098df760-9588-4719-9283-6dec0c8676e4","Type":"ContainerDied","Data":"8128b561fa1fabfd8a710ffc744153e07dc5ad8c5ece71556d3079a2090c3a30"} Nov 02 14:16:58.346092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:58.345869 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" event={"ID":"098df760-9588-4719-9283-6dec0c8676e4","Type":"ContainerStarted","Data":"915b48be3961a3f7417a23ab0996c4228ff985c40eb4fb943bb17c21f46fd377"} Nov 02 14:16:59.504668 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.504644 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:16:59.601826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.601772 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098df760-9588-4719-9283-6dec0c8676e4-kube-api-access-2lnrd\" (UniqueName: \"kubernetes.io/projected/098df760-9588-4719-9283-6dec0c8676e4-kube-api-access-2lnrd\") pod \"098df760-9588-4719-9283-6dec0c8676e4\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " Nov 02 14:16:59.602122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.602097 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098df760-9588-4719-9283-6dec0c8676e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098df760-9588-4719-9283-6dec0c8676e4-maas-api-openshift-ca\") pod \"098df760-9588-4719-9283-6dec0c8676e4\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " Nov 02 14:16:59.602257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.602241 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/098df760-9588-4719-9283-6dec0c8676e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098df760-9588-4719-9283-6dec0c8676e4-maas-api-xks-ca\") pod \"098df760-9588-4719-9283-6dec0c8676e4\" (UID: \"098df760-9588-4719-9283-6dec0c8676e4\") " Nov 02 14:16:59.603893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.603758 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098df760-9588-4719-9283-6dec0c8676e4-maas-api-openshift-ca" pod "098df760-9588-4719-9283-6dec0c8676e4" (UID: "098df760-9588-4719-9283-6dec0c8676e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:16:59.605411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.605384 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098df760-9588-4719-9283-6dec0c8676e4-maas-api-xks-ca" pod "098df760-9588-4719-9283-6dec0c8676e4" (UID: "098df760-9588-4719-9283-6dec0c8676e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:16:59.605834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.605810 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098df760-9588-4719-9283-6dec0c8676e4-kube-api-access-2lnrd" pod "098df760-9588-4719-9283-6dec0c8676e4" (UID: "098df760-9588-4719-9283-6dec0c8676e4"). InnerVolumeSpecName "kube-api-access-2lnrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:16:59.704276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.704246 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lnrd\" (UniqueName: \"kubernetes.io/projected/098df760-9588-4719-9283-6dec0c8676e4-kube-api-access-2lnrd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:59.704276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.704273 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/098df760-9588-4719-9283-6dec0c8676e4-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:16:59.704415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:16:59.704284 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/098df760-9588-4719-9283-6dec0c8676e4-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:00.038895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:00.038865 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96"] Nov 02 14:17:00.047346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:00.047310 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f0e314-f7m96"] Nov 02 14:17:00.362672 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:00.362581 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" event={"ID":"098df760-9588-4719-9283-6dec0c8676e4","Type":"ContainerDied","Data":"915b48be3961a3f7417a23ab0996c4228ff985c40eb4fb943bb17c21f46fd377"} Nov 02 14:17:00.362840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:00.362676 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915b48be3961a3f7417a23ab0996c4228ff985c40eb4fb943bb17c21f46fd377" Nov 02 14:17:00.362840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:00.362593 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj" Nov 02 14:17:01.036543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:01.036512 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q"] Nov 02 14:17:01.049492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:01.049463 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b654dd-pjg2q"] Nov 02 14:17:01.076338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:01.076308 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30cd348d-5e87-4114-a53c-ffb9095bd1d8" path="/var/lib/kubelet/pods/30cd348d-5e87-4114-a53c-ffb9095bd1d8/volumes" Nov 02 14:17:01.076822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:01.076801 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f58be5-07f2-4e8d-b619-98cb13afe2b5" path="/var/lib/kubelet/pods/61f58be5-07f2-4e8d-b619-98cb13afe2b5/volumes" Nov 02 14:17:04.033324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:04.033294 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7"] Nov 02 14:17:04.051680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:04.051638 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080004d2-8h5h7"] Nov 02 14:17:05.076255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:05.076225 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aaaa5a9-9ae2-4b85-83f4-46304acb5005" path="/var/lib/kubelet/pods/4aaaa5a9-9ae2-4b85-83f4-46304acb5005/volumes" Nov 02 14:17:07.078133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:07.078102 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:17:10.893508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:10.893473 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn"] Nov 02 14:17:10.904287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:10.904262 4431 cpu_manager.go:414] "containerMap: removing container" podUID="098df760-9588-4719-9283-6dec0c8676e4" containerName="revoke-keys" Nov 02 14:17:10.904287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:10.904284 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098df760-9588-4719-9283-6dec0c8676e4" containerName="revoke-keys" Nov 02 14:17:10.904454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:10.904413 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="098df760-9588-4719-9283-6dec0c8676e4" containerName="revoke-keys" Nov 02 14:17:10.908471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:10.908447 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:10.925768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:10.925740 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn"] Nov 02 14:17:10.931154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:10.931134 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f1e1c2-openshift-default-dockercfg-nn2q9\"" Nov 02 14:17:11.000866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.000836 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-envoy\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.001024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.000916 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-kube-api-access-9hv4t\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.001024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.000955 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-podinfo\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.001024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.000988 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-socket\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.001169 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.001044 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-credential-socket\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.001169 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.001067 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-token\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.001169 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.001142 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-certs\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.001291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.001188 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-data\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.001330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.001295 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istiod-ca-cert\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103095 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-kube-api-access-9hv4t\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103270 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-podinfo\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103403 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103337 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-socket\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103403 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103360 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-credential-socket\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103403 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103382 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-token\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103430 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-certs\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103520 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-data\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103683 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103619 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istiod-ca-cert\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103683 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103656 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-envoy\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.103937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.103913 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-credential-socket\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.104142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.104116 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-socket\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.104206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.104167 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-certs\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.104394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.104370 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-data\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.105039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.105014 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istiod-ca-cert\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.127891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.107089 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-envoy\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.127891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.117260 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-podinfo\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.135264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.135140 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-kube-api-access-9hv4t\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.144375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.144280 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-token\") pod \"e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.230260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.230007 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:11.343894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.343831 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz"] Nov 02 14:17:11.350217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.350190 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.377630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.377583 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz"] Nov 02 14:17:11.509481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509383 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.509481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509423 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.509481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509453 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.509768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509630 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.509768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509662 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.509902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509836 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdef57e5-d716-4090-96d0-eaa93c5fc089-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.509902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509883 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.510025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509905 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.510025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.509951 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9zc\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-kube-api-access-6m9zc\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.587174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.587143 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn"] Nov 02 14:17:11.589391 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:17:11.589361 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d3dd31_1898_4fdc_9f62_21d736f8ec14.slice/crio-6da70dc4e16390479b96e2fad158e74e86bb72dfe15471a7876dc6250acff8bc WatchSource:0}: Error finding container 6da70dc4e16390479b96e2fad158e74e86bb72dfe15471a7876dc6250acff8bc: Status 404 returned error can't find the container with id 6da70dc4e16390479b96e2fad158e74e86bb72dfe15471a7876dc6250acff8bc Nov 02 14:17:11.592570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.592535 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:11.592659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.592617 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:11.592730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.592664 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:11.610900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.610879 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.610961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.610909 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611052 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdef57e5-d716-4090-96d0-eaa93c5fc089-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611088 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611107 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611132 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9zc\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-kube-api-access-6m9zc\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611234 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611253 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611278 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611312 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611614 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611814 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdef57e5-d716-4090-96d0-eaa93c5fc089-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.611997 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.611935 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.612171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.612152 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.613981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.613938 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.614823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.614771 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.636311 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.636285 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.637722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.637698 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9zc\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-kube-api-access-6m9zc\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:11.682535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:11.682508 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:12.075227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.074691 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz"] Nov 02 14:17:12.101242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.101209 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:12.101334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.101275 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:12.101334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.101308 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:12.497160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.497111 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" event={"ID":"83d3dd31-1898-4fdc-9f62-21d736f8ec14","Type":"ContainerStarted","Data":"18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8"} Nov 02 14:17:12.497832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.497509 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" event={"ID":"83d3dd31-1898-4fdc-9f62-21d736f8ec14","Type":"ContainerStarted","Data":"6da70dc4e16390479b96e2fad158e74e86bb72dfe15471a7876dc6250acff8bc"} Nov 02 14:17:12.499503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.499474 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" event={"ID":"fdef57e5-d716-4090-96d0-eaa93c5fc089","Type":"ContainerStarted","Data":"09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae"} Nov 02 14:17:12.499635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.499524 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" event={"ID":"fdef57e5-d716-4090-96d0-eaa93c5fc089","Type":"ContainerStarted","Data":"4bf5231eca418beee95eb20292df9d7fd5962582a18daeacc0508a08e3d74e97"} Nov 02 14:17:12.550718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.550669 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" podStartSLOduration=2.550655967 podStartE2EDuration="2.550655967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:12.546136492 +0000 UTC m=+1197.583241428" watchObservedRunningTime="2026-11-02 14:17:12.550655967 +0000 UTC m=+1197.587760953" Nov 02 14:17:12.595469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.595420 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" podStartSLOduration=1.595407612 podStartE2EDuration="1.595407612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:12.593304612 +0000 UTC m=+1197.630409546" watchObservedRunningTime="2026-11-02 14:17:12.595407612 +0000 UTC m=+1197.632512543" Nov 02 14:17:12.685844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:12.683882 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:13.230827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.230791 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:13.236067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.236040 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:13.509396 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.509167 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:13.510436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.510416 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:13.696702 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.696665 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 14:17:13.696955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.696741 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 14:17:13.716821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.714518 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888"] Nov 02 14:17:13.724705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.724676 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.742482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.742454 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-866159\"/\"distinct-2-w3-866159-kserve-self-signed-certs\"" Nov 02 14:17:13.744992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.744915 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888"] Nov 02 14:17:13.848214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.848143 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-model-cache\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.848356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.848212 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fs5\" (UniqueName: \"kubernetes.io/projected/2d005e58-763a-4c16-98d9-fea871db9498-kube-api-access-t4fs5\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.848356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.848334 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-dshm\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.848437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.848359 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d005e58-763a-4c16-98d9-fea871db9498-tls-certs\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.848437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.848432 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-tmp-dir\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.848500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.848463 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-home\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.949818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.949744 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-model-cache\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.950395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.950366 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fs5\" (UniqueName: \"kubernetes.io/projected/2d005e58-763a-4c16-98d9-fea871db9498-kube-api-access-t4fs5\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.950542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.950497 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-dshm\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.950542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.950535 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d005e58-763a-4c16-98d9-fea871db9498-tls-certs\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.950667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.950597 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-tmp-dir\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.950667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.950642 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-home\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.951151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.950270 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-model-cache\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.951474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.951451 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-tmp-dir\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.952007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.951982 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-home\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.953673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.953650 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-dshm\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.956760 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.956737 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d005e58-763a-4c16-98d9-fea871db9498-tls-certs\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:13.979798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:13.978911 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fs5\" (UniqueName: \"kubernetes.io/projected/2d005e58-763a-4c16-98d9-fea871db9498-kube-api-access-t4fs5\") pod \"distinct-2-w3-866159-kserve-794b5d6cdd-bn888\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:14.043175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:14.043133 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:14.402050 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:14.402015 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888"] Nov 02 14:17:14.403044 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:17:14.403009 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d005e58_763a_4c16_98d9_fea871db9498.slice/crio-57dfc94617329e5159c95f38589539101f2f550345c28ad01de9e6f1a6d71e37 WatchSource:0}: Error finding container 57dfc94617329e5159c95f38589539101f2f550345c28ad01de9e6f1a6d71e37: Status 404 returned error can't find the container with id 57dfc94617329e5159c95f38589539101f2f550345c28ad01de9e6f1a6d71e37 Nov 02 14:17:14.520335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:14.520246 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" event={"ID":"2d005e58-763a-4c16-98d9-fea871db9498","Type":"ContainerStarted","Data":"5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2"} Nov 02 14:17:14.520335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:14.520292 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" event={"ID":"2d005e58-763a-4c16-98d9-fea871db9498","Type":"ContainerStarted","Data":"57dfc94617329e5159c95f38589539101f2f550345c28ad01de9e6f1a6d71e37"} Nov 02 14:17:14.567012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:14.566960 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" podStartSLOduration=1.5669466779999999 podStartE2EDuration="1.566946678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:14.564696907 +0000 UTC m=+1199.601801844" watchObservedRunningTime="2026-11-02 14:17:14.566946678 +0000 UTC m=+1199.604051610" Nov 02 14:17:14.684851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:14.683107 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 14:17:14.684851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:14.683162 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 14:17:15.036027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.035951 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm"] Nov 02 14:17:15.042315 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.042288 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b20bd8f-99srm"] Nov 02 14:17:15.044820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.044767 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:15.102991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.102962 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347c113b-213b-40f6-b92a-91005869198b" path="/var/lib/kubelet/pods/347c113b-213b-40f6-b92a-91005869198b/volumes" Nov 02 14:17:15.500696 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.500663 4431 scope.go:122] "RemoveContainer" containerID="7a79231295263092df4db19c86543759340902e6fe524aaaae940675c0d2e50a" Nov 02 14:17:15.511513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.511493 4431 scope.go:122] "RemoveContainer" containerID="77907e04e78c2e2661b4b2eccfcdca505e10613fb57fecc82c5f9731d10e52ed" Nov 02 14:17:15.521006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.520986 4431 scope.go:122] "RemoveContainer" containerID="6f435e57ddd07e7f6a17fbd4af0a544653b7efb54b02344fd8a2e2089d703033" Nov 02 14:17:15.531788 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.531761 4431 scope.go:122] "RemoveContainer" containerID="be72b20c94f13234c5c7131284916169fdc5684e6675fa646e631dda27333863" Nov 02 14:17:15.541468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.541451 4431 scope.go:122] "RemoveContainer" containerID="20530a7c908ad7943e48477eea749c93cba292a8a774bb3e8e4d12bfa5d21d7c" Nov 02 14:17:15.689800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.688172 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 14:17:15.689800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:15.688243 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 14:17:16.684802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:16.683084 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 14:17:16.684802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:16.683147 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 14:17:16.846797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:16.846706 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm"] Nov 02 14:17:16.852795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:16.847062 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" containerID="cri-o://46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0" gracePeriod=30 Nov 02 14:17:16.966261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:16.966164 4431 patch_prober.go:29] interesting pod/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:17:16.966561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:16.966521 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:17:17.741712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:17.720771 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 14:17:17.741712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:17.720882 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 14:17:18.683336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:18.683296 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 02 14:17:18.683504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:18.683398 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 02 14:17:19.715580 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:19.715546 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:19.716052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:19.715984 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:19.716989 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:19.716966 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:17:21.284728 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:17:21.284689 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:17:22.109506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.109485 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:17:22.158121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158099 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273ac84f-f3ab-44e7-998b-4fa269667ca0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273ac84f-f3ab-44e7-998b-4fa269667ca0-istiod-ca-cert\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158144 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-envoy\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158186 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-certs\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158274 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-credential-socket\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158435 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-token\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-token\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158467 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273ac84f-f3ab-44e7-998b-4fa269667ca0-istiod-ca-cert" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:22.158516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158473 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-kube-api-access-swnww\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-kube-api-access-swnww\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158505 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-socket\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158574 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-podinfo\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158573 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-certs" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:22.158682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158604 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-data\") pod \"273ac84f-f3ab-44e7-998b-4fa269667ca0\" (UID: \"273ac84f-f3ab-44e7-998b-4fa269667ca0\") " Nov 02 14:17:22.158682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158616 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-credential-socket" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:22.158944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.158808 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-socket" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:22.159717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.159187 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-data" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:22.159717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.159574 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.159717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.159599 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.159717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.159633 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.159717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.159655 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273ac84f-f3ab-44e7-998b-4fa269667ca0-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.159717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.159674 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.161091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.161064 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-envoy" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:22.161377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.161356 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-podinfo" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:17:22.161622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.161596 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-token" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:22.161712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.161668 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-kube-api-access-swnww" pod "273ac84f-f3ab-44e7-998b-4fa269667ca0" (UID: "273ac84f-f3ab-44e7-998b-4fa269667ca0"). InnerVolumeSpecName "kube-api-access-swnww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:22.260635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.260603 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.260635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.260630 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swnww\" (UniqueName: \"kubernetes.io/projected/273ac84f-f3ab-44e7-998b-4fa269667ca0-kube-api-access-swnww\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.260635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.260641 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.260886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.260649 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273ac84f-f3ab-44e7-998b-4fa269667ca0-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:22.613922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.613837 4431 generic.go:356] "Generic (PLEG): container finished" podID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerID="46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0" exitCode=0 Nov 02 14:17:22.613922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.613904 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" Nov 02 14:17:22.614443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.613928 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" event={"ID":"273ac84f-f3ab-44e7-998b-4fa269667ca0","Type":"ContainerDied","Data":"46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0"} Nov 02 14:17:22.614443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.613998 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm" event={"ID":"273ac84f-f3ab-44e7-998b-4fa269667ca0","Type":"ContainerDied","Data":"c4f40fe06a47468d136dcc45a12f4eed979d187bb9abeb6fe96ee7c5fa46f3c4"} Nov 02 14:17:22.614443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.614031 4431 scope.go:122] "RemoveContainer" containerID="46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0" Nov 02 14:17:22.635369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.635342 4431 scope.go:122] "RemoveContainer" containerID="3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b" Nov 02 14:17:22.645311 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.645289 4431 scope.go:122] "RemoveContainer" containerID="46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0" Nov 02 14:17:22.645570 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:17:22.645550 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0\": container with ID starting with 46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0 not found: ID does not exist" containerID="46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0" Nov 02 14:17:22.645647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.645582 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0"} err="failed to get container status \"46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0\": rpc error: code = NotFound desc = could not find container \"46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0\": container with ID starting with 46959d4caaa143b0c36957417fffecd3cf886604179e16bdb29a364a906188a0 not found: ID does not exist" Nov 02 14:17:22.645647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.645608 4431 scope.go:122] "RemoveContainer" containerID="3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b" Nov 02 14:17:22.645858 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:17:22.645841 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b\": container with ID starting with 3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b not found: ID does not exist" containerID="3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b" Nov 02 14:17:22.645902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.645865 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b"} err="failed to get container status \"3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b\": rpc error: code = NotFound desc = could not find container \"3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b\": container with ID starting with 3e59a10caae01e92dac0ae4e273a3c23f79ecfd266d4e240adc470f70d30727b not found: ID does not exist" Nov 02 14:17:22.674732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.674708 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm"] Nov 02 14:17:22.687025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:22.687001 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-de8cc7-openshift-default-6486dc96d5-b99sm"] Nov 02 14:17:23.076768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:23.076737 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" path="/var/lib/kubelet/pods/273ac84f-f3ab-44e7-998b-4fa269667ca0/volumes" Nov 02 14:17:24.044316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:24.044286 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:24.059959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:24.059932 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:24.641714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:24.641688 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:17:25.037946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:25.037876 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg"] Nov 02 14:17:25.041732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:25.041713 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0e4c6c-cmrtg"] Nov 02 14:17:25.078274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:25.078246 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b856e82-d30a-4729-a3ab-a45059046610" path="/var/lib/kubelet/pods/4b856e82-d30a-4729-a3ab-a45059046610/volumes" Nov 02 14:17:26.389936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.389903 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8"] Nov 02 14:17:26.390474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.390457 4431 cpu_manager.go:414] "containerMap: removing container" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" Nov 02 14:17:26.390474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.390474 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" Nov 02 14:17:26.390594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.390500 4431 cpu_manager.go:414] "containerMap: removing container" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" Nov 02 14:17:26.390594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.390506 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" Nov 02 14:17:26.390594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.390593 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" Nov 02 14:17:26.395960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.395937 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.422322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.422286 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8"] Nov 02 14:17:26.525401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525364 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666wv\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-kube-api-access-666wv\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.525401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525400 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.525658 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525421 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.525658 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525544 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.525739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525696 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.525870 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525853 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.525953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525898 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3052f4d4-e528-4e62-8b00-b48e23ce005a-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.525953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525925 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.526028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.525964 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.627407 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627381 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3052f4d4-e528-4e62-8b00-b48e23ce005a-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.627592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627422 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.627592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627440 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.627592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627475 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666wv\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-kube-api-access-666wv\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.627592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627495 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.627592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627512 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.627592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627530 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.628014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627648 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.628014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.627756 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.628014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.628000 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.628197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.628051 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.628197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.628161 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.628323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.628253 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3052f4d4-e528-4e62-8b00-b48e23ce005a-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.628323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.628283 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.630265 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.630243 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.630373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.630275 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.652492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.652431 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.652492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.652475 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-666wv\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-kube-api-access-666wv\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:26.717059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:26.717032 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:27.048970 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:17:27.048941 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3052f4d4_e528_4e62_8b00_b48e23ce005a.slice/crio-f70bc6e677383162aab33900c2ea2cf6f8a8d89bd9e56c63a5d0e50ab5f0f417 WatchSource:0}: Error finding container f70bc6e677383162aab33900c2ea2cf6f8a8d89bd9e56c63a5d0e50ab5f0f417: Status 404 returned error can't find the container with id f70bc6e677383162aab33900c2ea2cf6f8a8d89bd9e56c63a5d0e50ab5f0f417 Nov 02 14:17:27.049331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:27.049308 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8"] Nov 02 14:17:27.051424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:27.051395 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:27.051499 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:27.051455 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:27.051499 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:27.051485 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:27.655676 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:27.655644 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" event={"ID":"3052f4d4-e528-4e62-8b00-b48e23ce005a","Type":"ContainerStarted","Data":"e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13"} Nov 02 14:17:27.656078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:27.655700 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" event={"ID":"3052f4d4-e528-4e62-8b00-b48e23ce005a","Type":"ContainerStarted","Data":"f70bc6e677383162aab33900c2ea2cf6f8a8d89bd9e56c63a5d0e50ab5f0f417"} Nov 02 14:17:27.717270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:27.717192 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:27.724540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:27.724489 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" podStartSLOduration=1.7244749609999999 podStartE2EDuration="1.724474961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:27.719652063 +0000 UTC m=+1212.756756995" watchObservedRunningTime="2026-11-02 14:17:27.724474961 +0000 UTC m=+1212.761579893" Nov 02 14:17:28.719267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:28.719230 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:17:28.719683 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:28.719291 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:17:29.160970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:29.160938 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:17:29.724803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:29.719178 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:17:29.724803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:29.719240 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:17:30.728877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:30.727000 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:17:30.728877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:30.727076 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:17:31.573951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:31.573901 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:17:31.719801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:31.718251 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:17:31.719801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:31.718311 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:17:32.115801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:32.113835 4431 patch_prober.go:29] interesting pod/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:17:32.115801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:32.113896 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:17:32.717607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:32.717572 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:17:32.717824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:32.717636 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:17:33.719633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:33.719169 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 14:17:33.719633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:33.719237 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 14:17:34.017926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.015626 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx"] Nov 02 14:17:34.017926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.016708 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="273ac84f-f3ab-44e7-998b-4fa269667ca0" containerName="istio-proxy" Nov 02 14:17:34.028110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.028085 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.044506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.042503 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx"] Nov 02 14:17:34.047119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.047089 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:17:34.060837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.060811 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:17:34.116969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.116935 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.117140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.117087 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.117190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.117145 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69khz\" (UniqueName: \"kubernetes.io/projected/e14b5da6-876d-4af9-a5b3-7cb995708cca-kube-api-access-69khz\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.218396 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.218366 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.218580 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.218439 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69khz\" (UniqueName: \"kubernetes.io/projected/e14b5da6-876d-4af9-a5b3-7cb995708cca-kube-api-access-69khz\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.218580 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.218519 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.219099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.219074 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.221009 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.220985 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.243637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.243616 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69khz\" (UniqueName: \"kubernetes.io/projected/e14b5da6-876d-4af9-a5b3-7cb995708cca-kube-api-access-69khz\") pod \"maas-api-revoke-sub-4082dcca-d7hmx\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.350474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.350420 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:34.703533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.703507 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx"] Nov 02 14:17:34.704764 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:17:34.704740 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14b5da6_876d_4af9_a5b3_7cb995708cca.slice/crio-78afd969a3ef16ac6b30fe634d248ef502b825444b807fc9e89a6eb3d8177691 WatchSource:0}: Error finding container 78afd969a3ef16ac6b30fe634d248ef502b825444b807fc9e89a6eb3d8177691: Status 404 returned error can't find the container with id 78afd969a3ef16ac6b30fe634d248ef502b825444b807fc9e89a6eb3d8177691 Nov 02 14:17:34.895488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.895450 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:34.896851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.896826 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:34.898023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:34.897988 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:17:35.052734 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:35.052696 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" event={"ID":"e14b5da6-876d-4af9-a5b3-7cb995708cca","Type":"ContainerStarted","Data":"4c8add156fc79d688d49c62711a1f5409ab58df3aee1f7575f884ec083b2bb23"} Nov 02 14:17:35.053042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:35.052758 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" event={"ID":"e14b5da6-876d-4af9-a5b3-7cb995708cca","Type":"ContainerStarted","Data":"78afd969a3ef16ac6b30fe634d248ef502b825444b807fc9e89a6eb3d8177691"} Nov 02 14:17:35.066355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:35.066328 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf"] Nov 02 14:17:35.081063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:35.081033 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b68a13-8sttf"] Nov 02 14:17:35.104623 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:35.104572 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" podStartSLOduration=2.104562124 podStartE2EDuration="2.104562124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:35.103267023 +0000 UTC m=+1220.140371956" watchObservedRunningTime="2026-11-02 14:17:35.104562124 +0000 UTC m=+1220.141667055" Nov 02 14:17:36.063307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:36.063276 4431 generic.go:356] "Generic (PLEG): container finished" podID="e14b5da6-876d-4af9-a5b3-7cb995708cca" containerID="4c8add156fc79d688d49c62711a1f5409ab58df3aee1f7575f884ec083b2bb23" exitCode=0 Nov 02 14:17:36.063724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:36.063367 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" event={"ID":"e14b5da6-876d-4af9-a5b3-7cb995708cca","Type":"ContainerDied","Data":"4c8add156fc79d688d49c62711a1f5409ab58df3aee1f7575f884ec083b2bb23"} Nov 02 14:17:37.076214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.076181 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938f348b-7d83-4670-b488-eb500446e69d" path="/var/lib/kubelet/pods/938f348b-7d83-4670-b488-eb500446e69d/volumes" Nov 02 14:17:37.213280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.213257 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:37.258666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.258643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-xks-ca\") pod \"e14b5da6-876d-4af9-a5b3-7cb995708cca\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " Nov 02 14:17:37.258868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.258689 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14b5da6-876d-4af9-a5b3-7cb995708cca-kube-api-access-69khz\" (UniqueName: \"kubernetes.io/projected/e14b5da6-876d-4af9-a5b3-7cb995708cca-kube-api-access-69khz\") pod \"e14b5da6-876d-4af9-a5b3-7cb995708cca\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " Nov 02 14:17:37.258868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.258737 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-openshift-ca\") pod \"e14b5da6-876d-4af9-a5b3-7cb995708cca\" (UID: \"e14b5da6-876d-4af9-a5b3-7cb995708cca\") " Nov 02 14:17:37.259711 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.259681 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-openshift-ca" pod "e14b5da6-876d-4af9-a5b3-7cb995708cca" (UID: "e14b5da6-876d-4af9-a5b3-7cb995708cca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:37.261610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.261580 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14b5da6-876d-4af9-a5b3-7cb995708cca-kube-api-access-69khz" pod "e14b5da6-876d-4af9-a5b3-7cb995708cca" (UID: "e14b5da6-876d-4af9-a5b3-7cb995708cca"). InnerVolumeSpecName "kube-api-access-69khz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:37.261610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.261601 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-xks-ca" pod "e14b5da6-876d-4af9-a5b3-7cb995708cca" (UID: "e14b5da6-876d-4af9-a5b3-7cb995708cca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:17:37.360296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.360231 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:37.360296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.360258 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69khz\" (UniqueName: \"kubernetes.io/projected/e14b5da6-876d-4af9-a5b3-7cb995708cca-kube-api-access-69khz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:37.360296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:37.360275 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e14b5da6-876d-4af9-a5b3-7cb995708cca-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:38.087544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:38.087512 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" event={"ID":"e14b5da6-876d-4af9-a5b3-7cb995708cca","Type":"ContainerDied","Data":"78afd969a3ef16ac6b30fe634d248ef502b825444b807fc9e89a6eb3d8177691"} Nov 02 14:17:38.088049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:38.087582 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78afd969a3ef16ac6b30fe634d248ef502b825444b807fc9e89a6eb3d8177691" Nov 02 14:17:38.088049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:38.087518 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx" Nov 02 14:17:38.090562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:38.090539 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4"] Nov 02 14:17:38.102745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:38.102721 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442194a2-tstn4"] Nov 02 14:17:39.075747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:39.075712 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641b4904-b6de-421c-b5a3-c6ab2a5fc006" path="/var/lib/kubelet/pods/641b4904-b6de-421c-b5a3-c6ab2a5fc006/volumes" Nov 02 14:17:41.042074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:41.042042 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4"] Nov 02 14:17:41.054413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:41.054386 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f1b84cb-h5qd4"] Nov 02 14:17:41.076308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:41.076276 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fa5579-2bd0-4be1-91b7-b7c71e00d214" path="/var/lib/kubelet/pods/f7fa5579-2bd0-4be1-91b7-b7c71e00d214/volumes" Nov 02 14:17:44.036562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:44.036524 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925"] Nov 02 14:17:44.049700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:44.049661 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b8ef5d-2x925"] Nov 02 14:17:45.153813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:45.150684 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1a733b-05ec-4aad-8ebc-f116c54580e4" path="/var/lib/kubelet/pods/1d1a733b-05ec-4aad-8ebc-f116c54580e4/volumes" Nov 02 14:17:46.697557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.697512 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv"] Nov 02 14:17:46.698373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.698352 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e14b5da6-876d-4af9-a5b3-7cb995708cca" containerName="revoke-keys" Nov 02 14:17:46.698373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.698373 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14b5da6-876d-4af9-a5b3-7cb995708cca" containerName="revoke-keys" Nov 02 14:17:46.698526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.698498 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14b5da6-876d-4af9-a5b3-7cb995708cca" containerName="revoke-keys" Nov 02 14:17:46.703409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.703388 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv"] Nov 02 14:17:46.703550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.703476 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.719751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.719733 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:17:46.733073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.733053 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:17:46.754495 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.754473 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.754599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.754525 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkhlf\" (UniqueName: \"kubernetes.io/projected/58577ac1-c47c-4730-8c15-688c84f0e39a-kube-api-access-jkhlf\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.754759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.754737 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.856326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.856292 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkhlf\" (UniqueName: \"kubernetes.io/projected/58577ac1-c47c-4730-8c15-688c84f0e39a-kube-api-access-jkhlf\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.856506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.856491 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.856601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.856587 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.857283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.857242 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.859066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.859045 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:46.897307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:46.897277 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkhlf\" (UniqueName: \"kubernetes.io/projected/58577ac1-c47c-4730-8c15-688c84f0e39a-kube-api-access-jkhlf\") pod \"maas-api-revoke-sub-333805c3-wb4mv\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:47.018689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:47.018623 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:47.351626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:47.351601 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv"] Nov 02 14:17:47.353182 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:17:47.353149 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58577ac1_c47c_4730_8c15_688c84f0e39a.slice/crio-f6f1f0d337f0f96984fe7b609b85c6961c4570618eba7e656f53446d3bf5c498 WatchSource:0}: Error finding container f6f1f0d337f0f96984fe7b609b85c6961c4570618eba7e656f53446d3bf5c498: Status 404 returned error can't find the container with id f6f1f0d337f0f96984fe7b609b85c6961c4570618eba7e656f53446d3bf5c498 Nov 02 14:17:48.339832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:48.332941 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" event={"ID":"58577ac1-c47c-4730-8c15-688c84f0e39a","Type":"ContainerStarted","Data":"ecb5fe7f39b9cf0e7facbea1b6194b6e8ead6b9cc8755377b281a8b92841b577"} Nov 02 14:17:48.339832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:48.333015 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" event={"ID":"58577ac1-c47c-4730-8c15-688c84f0e39a","Type":"ContainerStarted","Data":"f6f1f0d337f0f96984fe7b609b85c6961c4570618eba7e656f53446d3bf5c498"} Nov 02 14:17:48.396814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:48.396520 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" podStartSLOduration=2.3965077949999998 podStartE2EDuration="2.396507795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:48.396072963 +0000 UTC m=+1233.433177897" watchObservedRunningTime="2026-11-02 14:17:48.396507795 +0000 UTC m=+1233.433612726" Nov 02 14:17:49.036274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:49.036241 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v"] Nov 02 14:17:49.040431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:49.040403 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2baee653-t8l9v"] Nov 02 14:17:49.101541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:49.101502 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db8ce96-a4c2-4be2-ab06-82ebfaef2708" path="/var/lib/kubelet/pods/1db8ce96-a4c2-4be2-ab06-82ebfaef2708/volumes" Nov 02 14:17:49.343605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:49.343517 4431 generic.go:356] "Generic (PLEG): container finished" podID="58577ac1-c47c-4730-8c15-688c84f0e39a" containerID="ecb5fe7f39b9cf0e7facbea1b6194b6e8ead6b9cc8755377b281a8b92841b577" exitCode=0 Nov 02 14:17:49.344030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:49.343732 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" event={"ID":"58577ac1-c47c-4730-8c15-688c84f0e39a","Type":"ContainerDied","Data":"ecb5fe7f39b9cf0e7facbea1b6194b6e8ead6b9cc8755377b281a8b92841b577"} Nov 02 14:17:50.653642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.653354 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:50.779179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.779145 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-xks-ca\") pod \"58577ac1-c47c-4730-8c15-688c84f0e39a\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " Nov 02 14:17:50.779359 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.779195 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-openshift-ca\") pod \"58577ac1-c47c-4730-8c15-688c84f0e39a\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " Nov 02 14:17:50.779419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.779397 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58577ac1-c47c-4730-8c15-688c84f0e39a-kube-api-access-jkhlf\" (UniqueName: \"kubernetes.io/projected/58577ac1-c47c-4730-8c15-688c84f0e39a-kube-api-access-jkhlf\") pod \"58577ac1-c47c-4730-8c15-688c84f0e39a\" (UID: \"58577ac1-c47c-4730-8c15-688c84f0e39a\") " Nov 02 14:17:50.779645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.779612 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-openshift-ca" pod "58577ac1-c47c-4730-8c15-688c84f0e39a" (UID: "58577ac1-c47c-4730-8c15-688c84f0e39a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:50.780022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.779982 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:50.782140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.782115 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58577ac1-c47c-4730-8c15-688c84f0e39a-kube-api-access-jkhlf" pod "58577ac1-c47c-4730-8c15-688c84f0e39a" (UID: "58577ac1-c47c-4730-8c15-688c84f0e39a"). InnerVolumeSpecName "kube-api-access-jkhlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:50.783190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.783161 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-xks-ca" pod "58577ac1-c47c-4730-8c15-688c84f0e39a" (UID: "58577ac1-c47c-4730-8c15-688c84f0e39a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:17:50.881370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.881287 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58577ac1-c47c-4730-8c15-688c84f0e39a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:50.881370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:50.881330 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkhlf\" (UniqueName: \"kubernetes.io/projected/58577ac1-c47c-4730-8c15-688c84f0e39a-kube-api-access-jkhlf\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:17:51.286837 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:17:51.286796 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:17:51.365751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:51.365715 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" event={"ID":"58577ac1-c47c-4730-8c15-688c84f0e39a","Type":"ContainerDied","Data":"f6f1f0d337f0f96984fe7b609b85c6961c4570618eba7e656f53446d3bf5c498"} Nov 02 14:17:51.366007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:51.365867 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv" Nov 02 14:17:51.366584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:51.366532 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6f1f0d337f0f96984fe7b609b85c6961c4570618eba7e656f53446d3bf5c498" Nov 02 14:17:53.997012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:53.996983 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77"] Nov 02 14:17:53.997429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:53.997165 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" containerID="cri-o://517d801f9611c704e27598d6d6acad4c219de62db7688119b14b0a7106bf87dc" gracePeriod=30 Nov 02 14:17:54.010206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.010179 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc"] Nov 02 14:17:54.010378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.010358 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" containerID="cri-o://bd7aed879dd2f5904a5abff168082320c38dc5fe89d3c805fe5b56a3c20ebb57" gracePeriod=30 Nov 02 14:17:54.014107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.014085 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d"] Nov 02 14:17:54.014255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.014238 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" containerID="cri-o://106e7f7ec102bd39c9e8ac3355e5ba6080b1a9302a788fee182fe198212b00b6" gracePeriod=30 Nov 02 14:17:54.023585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.023563 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq"] Nov 02 14:17:54.023758 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.023736 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" containerID="cri-o://19702f4521c1064ea290389feeb0ea22b3456c0c2db8e27584e3d3c032cbb409" gracePeriod=30 Nov 02 14:17:54.417906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.415645 4431 generic.go:356] "Generic (PLEG): container finished" podID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerID="18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8" exitCode=0 Nov 02 14:17:54.417906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.415830 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" event={"ID":"83d3dd31-1898-4fdc-9f62-21d736f8ec14","Type":"ContainerDied","Data":"18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8"} Nov 02 14:17:54.417906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.416349 4431 scope.go:122] "RemoveContainer" containerID="18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8" Nov 02 14:17:54.418416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.418101 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:54.418416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.418192 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:54.418416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:54.418225 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:55.425789 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:55.425754 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" event={"ID":"83d3dd31-1898-4fdc-9f62-21d736f8ec14","Type":"ContainerStarted","Data":"243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d"} Nov 02 14:17:56.231133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.231105 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:56.232335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.232307 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 14:17:56.232468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.232354 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 14:17:56.536288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.536195 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb"] Nov 02 14:17:56.537014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.536991 4431 cpu_manager.go:414] "containerMap: removing container" podUID="58577ac1-c47c-4730-8c15-688c84f0e39a" containerName="revoke-keys" Nov 02 14:17:56.537014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.537014 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58577ac1-c47c-4730-8c15-688c84f0e39a" containerName="revoke-keys" Nov 02 14:17:56.537199 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.537184 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="58577ac1-c47c-4730-8c15-688c84f0e39a" containerName="revoke-keys" Nov 02 14:17:56.541285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.540391 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.598685 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.598648 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb"] Nov 02 14:17:56.654881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.654849 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.655050 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.654896 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.655050 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.654980 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.655170 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.655140 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8cn\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-kube-api-access-9b8cn\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.655223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.655187 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.655300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.655288 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.655340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.655317 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.655394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.655339 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.655458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.655444 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.756963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.756925 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.756971 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757056 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8cn\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-kube-api-access-9b8cn\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757091 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757146 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757188 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757224 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757288 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757366 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-credential-socket\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757290 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757486 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757546 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-certs\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.757827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.757771 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-data\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.758225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.758197 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istiod-ca-cert\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.759612 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.759592 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-podinfo\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.759752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.759732 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-envoy\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.792897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.791681 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-token\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.806927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.806834 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8cn\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-kube-api-access-9b8cn\") pod \"e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:56.874311 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:56.874023 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:57.232392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:57.231918 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 14:17:57.232392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:57.231988 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 14:17:57.505470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:57.505440 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb"] Nov 02 14:17:57.512204 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:17:57.511750 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ccc3a9_7f72_467c_a8dd_b4783c6c08f4.slice/crio-ac1d056caa24de8fa6931c7dd627fc301aa6e0a842b338a79179ba345f797de5 WatchSource:0}: Error finding container ac1d056caa24de8fa6931c7dd627fc301aa6e0a842b338a79179ba345f797de5: Status 404 returned error can't find the container with id ac1d056caa24de8fa6931c7dd627fc301aa6e0a842b338a79179ba345f797de5 Nov 02 14:17:57.514757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:57.514714 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:57.514898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:57.514849 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:57.514945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:57.514895 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:17:58.234159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.234126 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 14:17:58.234669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.234221 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 14:17:58.453449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.453415 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" event={"ID":"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4","Type":"ContainerStarted","Data":"0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82"} Nov 02 14:17:58.453625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.453468 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" event={"ID":"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4","Type":"ContainerStarted","Data":"ac1d056caa24de8fa6931c7dd627fc301aa6e0a842b338a79179ba345f797de5"} Nov 02 14:17:58.520857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.519756 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:17:58.540925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.540879 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" podStartSLOduration=2.540864109 podStartE2EDuration="2.540864109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:58.526255767 +0000 UTC m=+1243.563360700" watchObservedRunningTime="2026-11-02 14:17:58.540864109 +0000 UTC m=+1243.577969040" Nov 02 14:17:58.881868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.878979 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:17:58.895799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.893976 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 14:17:58.895799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:58.894041 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 14:17:59.064443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.064393 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm"] Nov 02 14:17:59.080756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.080725 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ede89e-n4kqm"] Nov 02 14:17:59.231018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.230983 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 14:17:59.231203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.231057 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 14:17:59.469326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.469282 4431 generic.go:356] "Generic (PLEG): container finished" podID="59953df1-7268-4846-a4af-31361b224205" containerID="106e7f7ec102bd39c9e8ac3355e5ba6080b1a9302a788fee182fe198212b00b6" exitCode=0 Nov 02 14:17:59.469643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.469357 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" event={"ID":"59953df1-7268-4846-a4af-31361b224205","Type":"ContainerDied","Data":"106e7f7ec102bd39c9e8ac3355e5ba6080b1a9302a788fee182fe198212b00b6"} Nov 02 14:17:59.518825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.508843 4431 generic.go:356] "Generic (PLEG): container finished" podID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerID="517d801f9611c704e27598d6d6acad4c219de62db7688119b14b0a7106bf87dc" exitCode=0 Nov 02 14:17:59.519014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.518914 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" event={"ID":"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d","Type":"ContainerDied","Data":"517d801f9611c704e27598d6d6acad4c219de62db7688119b14b0a7106bf87dc"} Nov 02 14:17:59.561222 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.561180 4431 generic.go:356] "Generic (PLEG): container finished" podID="10df52e7-8bdd-444d-8942-0df2310259cd" containerID="19702f4521c1064ea290389feeb0ea22b3456c0c2db8e27584e3d3c032cbb409" exitCode=0 Nov 02 14:17:59.561404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.561329 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" event={"ID":"10df52e7-8bdd-444d-8942-0df2310259cd","Type":"ContainerDied","Data":"19702f4521c1064ea290389feeb0ea22b3456c0c2db8e27584e3d3c032cbb409"} Nov 02 14:17:59.564197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.564159 4431 generic.go:356] "Generic (PLEG): container finished" podID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerID="bd7aed879dd2f5904a5abff168082320c38dc5fe89d3c805fe5b56a3c20ebb57" exitCode=0 Nov 02 14:17:59.575092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.565327 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" event={"ID":"e59f3098-fd86-4ed3-ae76-12e47eaf4e85","Type":"ContainerDied","Data":"bd7aed879dd2f5904a5abff168082320c38dc5fe89d3c805fe5b56a3c20ebb57"} Nov 02 14:17:59.737409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.737385 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:17:59.743074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.743049 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:17:59.749525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.748743 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:17:59.753285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.753265 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910324 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-kube-api-access-ffgd8\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-kube-api-access-ffgd8\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910432 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-credential-socket\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910474 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-credential-socket\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910505 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-podinfo\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910553 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-credential-socket\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910584 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-socket\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910619 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-certs\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910646 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-certs\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910716 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-kube-api-access-7trhk\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-kube-api-access-7trhk\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.910798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.910759 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-certs\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.915813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.911536 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-envoy\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.915813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.911595 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-data\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.915813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.911645 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-podinfo\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.915813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.911715 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-kube-api-access-l5f7q\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-kube-api-access-l5f7q\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.915813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.911745 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-token\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-token\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916217 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-envoy\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916278 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-socket\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916308 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-token\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-token\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916354 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-envoy\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916382 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istiod-ca-cert\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916414 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-certs\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916458 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-data\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916503 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-socket\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916536 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-data\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916563 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-istio-token\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-istio-token\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916601 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59953df1-7268-4846-a4af-31361b224205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59953df1-7268-4846-a4af-31361b224205-istiod-ca-cert\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916647 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10df52e7-8bdd-444d-8942-0df2310259cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df52e7-8bdd-444d-8942-0df2310259cd-istio-podinfo\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916722 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59953df1-7268-4846-a4af-31361b224205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59953df1-7268-4846-a4af-31361b224205-istio-podinfo\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916761 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-credential-socket\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916807 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-istio-token\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-istio-token\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916898 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-socket\") pod \"59953df1-7268-4846-a4af-31361b224205\" (UID: \"59953df1-7268-4846-a4af-31361b224205\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.916969 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-data\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.917012 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-envoy\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.917084 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10df52e7-8bdd-444d-8942-0df2310259cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df52e7-8bdd-444d-8942-0df2310259cd-istiod-ca-cert\") pod \"10df52e7-8bdd-444d-8942-0df2310259cd\" (UID: \"10df52e7-8bdd-444d-8942-0df2310259cd\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.917116 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istiod-ca-cert\") pod \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\" (UID: \"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.917183 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-kube-api-access-wrnwt\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-kube-api-access-wrnwt\") pod \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\" (UID: \"e59f3098-fd86-4ed3-ae76-12e47eaf4e85\") " Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.926052 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-credential-socket" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.926265 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-credential-socket" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.927222 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-kube-api-access-wrnwt" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "kube-api-access-wrnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.927479 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-kube-api-access-ffgd8" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "kube-api-access-ffgd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.927555 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istiod-ca-cert" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:59.927798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.927656 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-data" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.927897 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-credential-socket" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.927938 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.927984 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.928091 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-socket" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.928264 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-certs" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.928509 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-certs" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.928635 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59953df1-7268-4846-a4af-31361b224205-istiod-ca-cert" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.928974 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-certs" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.929314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.929176 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-data" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.929863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.929468 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-socket" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.929863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.929574 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-data" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.930334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.930170 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-data" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.931089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.930807 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-credential-socket" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.931458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.931212 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10df52e7-8bdd-444d-8942-0df2310259cd-istiod-ca-cert" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:59.931926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.931751 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istiod-ca-cert" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:17:59.932014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.931927 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-socket" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.932359 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.932218 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-socket" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.933121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.933094 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-certs" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.933666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.933634 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-istio-token" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:59.934180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.933930 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-podinfo" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:17:59.934180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.933992 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10df52e7-8bdd-444d-8942-0df2310259cd-istio-podinfo" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:17:59.934180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.934041 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-envoy" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.934180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.934062 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-podinfo" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:17:59.934180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.934152 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-istio-token" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:59.934596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.934565 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-kube-api-access-l5f7q" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "kube-api-access-l5f7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:59.935979 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.935954 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-token" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:59.936101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.936079 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-envoy" pod "10df52e7-8bdd-444d-8942-0df2310259cd" (UID: "10df52e7-8bdd-444d-8942-0df2310259cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.936293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.936229 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-envoy" pod "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" (UID: "3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.936293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.936236 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-kube-api-access-7trhk" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "kube-api-access-7trhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:17:59.936414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.936377 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-envoy" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:17:59.936500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.936468 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59953df1-7268-4846-a4af-31361b224205-istio-podinfo" pod "59953df1-7268-4846-a4af-31361b224205" (UID: "59953df1-7268-4846-a4af-31361b224205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:17:59.936648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:17:59.936634 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-token" pod "e59f3098-fd86-4ed3-ae76-12e47eaf4e85" (UID: "e59f3098-fd86-4ed3-ae76-12e47eaf4e85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019013 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10df52e7-8bdd-444d-8942-0df2310259cd-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019050 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019068 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnwt\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-kube-api-access-wrnwt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019083 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgd8\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-kube-api-access-ffgd8\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019098 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019112 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019126 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019139 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019153 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019166 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019179 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019192 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trhk\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-kube-api-access-7trhk\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019205 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019219 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019231 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019244 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019256 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5f7q\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-kube-api-access-l5f7q\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019269 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019282 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019296 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019312 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019325 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019337 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019351 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019363 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019375 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019388 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019402 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59953df1-7268-4846-a4af-31361b224205-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019415 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59953df1-7268-4846-a4af-31361b224205-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019427 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10df52e7-8bdd-444d-8942-0df2310259cd-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019441 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59953df1-7268-4846-a4af-31361b224205-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019453 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019466 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10df52e7-8bdd-444d-8942-0df2310259cd-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019483 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59953df1-7268-4846-a4af-31361b224205-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019495 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10df52e7-8bdd-444d-8942-0df2310259cd-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.019864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.019507 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59f3098-fd86-4ed3-ae76-12e47eaf4e85-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:00.230815 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.230758 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 02 14:18:00.230984 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.230839 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 02 14:18:00.581521 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.581471 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" event={"ID":"10df52e7-8bdd-444d-8942-0df2310259cd","Type":"ContainerDied","Data":"a3f9ca44ea86b23a0d173f424d7ec2347fa440e1ab4c7de99807f41f00de452b"} Nov 02 14:18:00.583545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.583525 4431 scope.go:122] "RemoveContainer" containerID="19702f4521c1064ea290389feeb0ea22b3456c0c2db8e27584e3d3c032cbb409" Nov 02 14:18:00.584166 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.582643 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq" Nov 02 14:18:00.586114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.585024 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" event={"ID":"e59f3098-fd86-4ed3-ae76-12e47eaf4e85","Type":"ContainerDied","Data":"167454f2eb76ffd8f1b4c14d302ad7ddbdef707e4b6158f543e31579f10acb4f"} Nov 02 14:18:00.586217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.585092 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc" Nov 02 14:18:00.601614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.601581 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" event={"ID":"59953df1-7268-4846-a4af-31361b224205","Type":"ContainerDied","Data":"1d893194881486747c240cab8e5b6794d970784279447ed0e7cf88909a7df9ae"} Nov 02 14:18:00.601917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.601887 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d" Nov 02 14:18:00.604845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.604769 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" Nov 02 14:18:00.604965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.604906 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77" event={"ID":"3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d","Type":"ContainerDied","Data":"258189f925880a2b613ef0ed62c74af1a674827d17f9fc5772e31e6c13d2074b"} Nov 02 14:18:00.621004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.620952 4431 scope.go:122] "RemoveContainer" containerID="bd7aed879dd2f5904a5abff168082320c38dc5fe89d3c805fe5b56a3c20ebb57" Nov 02 14:18:00.631638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.631617 4431 scope.go:122] "RemoveContainer" containerID="106e7f7ec102bd39c9e8ac3355e5ba6080b1a9302a788fee182fe198212b00b6" Nov 02 14:18:00.644293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.644267 4431 scope.go:122] "RemoveContainer" containerID="517d801f9611c704e27598d6d6acad4c219de62db7688119b14b0a7106bf87dc" Nov 02 14:18:00.694797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.693835 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d"] Nov 02 14:18:00.703068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.703038 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-x2w9d"] Nov 02 14:18:00.720157 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.720128 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77"] Nov 02 14:18:00.724426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.724403 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-jxt77"] Nov 02 14:18:00.761346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.761314 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq"] Nov 02 14:18:00.765987 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.765964 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-fqjwq"] Nov 02 14:18:00.819741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.819712 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc"] Nov 02 14:18:00.823505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.823484 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-555d1e-openshift-default-7dd9fbcb87-86vvc"] Nov 02 14:18:00.890809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.890108 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 14:18:00.890809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:00.890160 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 14:18:01.098282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.098252 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" path="/var/lib/kubelet/pods/10df52e7-8bdd-444d-8942-0df2310259cd/volumes" Nov 02 14:18:01.098717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.098701 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" path="/var/lib/kubelet/pods/3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d/volumes" Nov 02 14:18:01.099178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.099160 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59953df1-7268-4846-a4af-31361b224205" path="/var/lib/kubelet/pods/59953df1-7268-4846-a4af-31361b224205/volumes" Nov 02 14:18:01.099868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.099853 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ea4292-a174-412e-9777-e33b9d197ac5" path="/var/lib/kubelet/pods/93ea4292-a174-412e-9777-e33b9d197ac5/volumes" Nov 02 14:18:01.100263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.100250 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" path="/var/lib/kubelet/pods/e59f3098-fd86-4ed3-ae76-12e47eaf4e85/volumes" Nov 02 14:18:01.235889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.235803 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:18:01.236875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.236856 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:18:01.875801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.875121 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 14:18:01.875801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:01.875177 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 14:18:02.037675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:02.037643 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x"] Nov 02 14:18:02.050197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:02.050164 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249e699c-2cf4x"] Nov 02 14:18:02.886799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:02.881154 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 14:18:02.886799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:02.881218 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 14:18:03.101338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.101293 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b428bd72-4fb6-4a99-96dc-b95287d36e8b" path="/var/lib/kubelet/pods/b428bd72-4fb6-4a99-96dc-b95287d36e8b/volumes" Nov 02 14:18:03.235709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.235628 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5"] Nov 02 14:18:03.236440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236420 4431 cpu_manager.go:414] "containerMap: removing container" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" Nov 02 14:18:03.236502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236444 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" Nov 02 14:18:03.236502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236494 4431 cpu_manager.go:414] "containerMap: removing container" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" Nov 02 14:18:03.236581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236501 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" Nov 02 14:18:03.236581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236518 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" Nov 02 14:18:03.236581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236525 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" Nov 02 14:18:03.236581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236546 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" Nov 02 14:18:03.236581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236552 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" Nov 02 14:18:03.236728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236673 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59f3098-fd86-4ed3-ae76-12e47eaf4e85" containerName="istio-proxy" Nov 02 14:18:03.236728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236692 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="59953df1-7268-4846-a4af-31361b224205" containerName="istio-proxy" Nov 02 14:18:03.236728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236708 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="10df52e7-8bdd-444d-8942-0df2310259cd" containerName="istio-proxy" Nov 02 14:18:03.236728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.236719 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c954aaf-4cb0-4b2b-86ed-77b79fd50c0d" containerName="istio-proxy" Nov 02 14:18:03.240771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.240727 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:03.255506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.255468 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b0080d-e55e-437f-b649-ec4578a9699b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5\" (UID: \"b5b0080d-e55e-437f-b649-ec4578a9699b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:03.255700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.255681 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqf5h\" (UniqueName: \"kubernetes.io/projected/b5b0080d-e55e-437f-b649-ec4578a9699b-kube-api-access-bqf5h\") pod \"payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5\" (UID: \"b5b0080d-e55e-437f-b649-ec4578a9699b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:03.263263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.263243 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5"] Nov 02 14:18:03.286799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.281862 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-623b4b-dockercfg-k2bp9\"" Nov 02 14:18:03.286799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.281932 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-623b4b\"" Nov 02 14:18:03.362905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.362872 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r"] Nov 02 14:18:03.364834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.363900 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqf5h\" (UniqueName: \"kubernetes.io/projected/b5b0080d-e55e-437f-b649-ec4578a9699b-kube-api-access-bqf5h\") pod \"payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5\" (UID: \"b5b0080d-e55e-437f-b649-ec4578a9699b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:03.365052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.364987 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b0080d-e55e-437f-b649-ec4578a9699b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5\" (UID: \"b5b0080d-e55e-437f-b649-ec4578a9699b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:03.365963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.365934 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b0080d-e55e-437f-b649-ec4578a9699b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5\" (UID: \"b5b0080d-e55e-437f-b649-ec4578a9699b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:03.390108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.390072 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:03.415619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.415038 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r"] Nov 02 14:18:03.422859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.422747 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqf5h\" (UniqueName: \"kubernetes.io/projected/b5b0080d-e55e-437f-b649-ec4578a9699b-kube-api-access-bqf5h\") pod \"payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5\" (UID: \"b5b0080d-e55e-437f-b649-ec4578a9699b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:03.466446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.466414 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r\" (UID: \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:03.466682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.466665 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9gnv\" (UniqueName: \"kubernetes.io/projected/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-kube-api-access-t9gnv\") pod \"payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r\" (UID: \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:03.553658 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.553576 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:03.568322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.568283 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r\" (UID: \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:03.568546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.568527 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9gnv\" (UniqueName: \"kubernetes.io/projected/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-kube-api-access-t9gnv\") pod \"payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r\" (UID: \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:03.569019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.568996 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r\" (UID: \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:03.599079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.599047 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9gnv\" (UniqueName: \"kubernetes.io/projected/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-kube-api-access-t9gnv\") pod \"payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r\" (UID: \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:03.750888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.750852 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:03.875049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.875014 4431 patch_prober.go:29] interesting pod/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 14:18:03.875196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.875072 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 14:18:03.898244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.898214 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5"] Nov 02 14:18:03.898646 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:03.898414 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b0080d_e55e_437f_b649_ec4578a9699b.slice/crio-d14f6234bfde2da8b9ded1db1222e3f3674074ed0ef4a96b30df49c80039de25 WatchSource:0}: Error finding container d14f6234bfde2da8b9ded1db1222e3f3674074ed0ef4a96b30df49c80039de25: Status 404 returned error can't find the container with id d14f6234bfde2da8b9ded1db1222e3f3674074ed0ef4a96b30df49c80039de25 Nov 02 14:18:03.900538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:03.900520 4431 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:18:04.072730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:04.072701 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r"] Nov 02 14:18:04.072933 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:04.072793 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01f4c12_63f6_431c_a9ca_a1205dd8aacf.slice/crio-9bd7042fa7a7fc50f291ad336dc0aafe62ef3a63d6cea82b4c0492cc96f8d2e1 WatchSource:0}: Error finding container 9bd7042fa7a7fc50f291ad336dc0aafe62ef3a63d6cea82b4c0492cc96f8d2e1: Status 404 returned error can't find the container with id 9bd7042fa7a7fc50f291ad336dc0aafe62ef3a63d6cea82b4c0492cc96f8d2e1 Nov 02 14:18:04.665745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:04.665706 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" event={"ID":"b01f4c12-63f6-431c-a9ca-a1205dd8aacf","Type":"ContainerStarted","Data":"9bd7042fa7a7fc50f291ad336dc0aafe62ef3a63d6cea82b4c0492cc96f8d2e1"} Nov 02 14:18:04.667581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:04.667548 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" event={"ID":"b5b0080d-e55e-437f-b649-ec4578a9699b","Type":"ContainerStarted","Data":"d14f6234bfde2da8b9ded1db1222e3f3674074ed0ef4a96b30df49c80039de25"} Nov 02 14:18:05.441315 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:05.440165 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:18:05.441956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:05.441296 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:18:05.442070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:05.442053 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:18:05.640610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:05.640581 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:18:07.726826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:07.719168 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" event={"ID":"b01f4c12-63f6-431c-a9ca-a1205dd8aacf","Type":"ContainerStarted","Data":"ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e"} Nov 02 14:18:07.726826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:07.722108 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" event={"ID":"b5b0080d-e55e-437f-b649-ec4578a9699b","Type":"ContainerStarted","Data":"0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0"} Nov 02 14:18:07.726826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:07.722744 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:07.726826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:07.722915 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:07.778836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:07.775866 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" podStartSLOduration=1.952468347 podStartE2EDuration="4.775842971s" totalImagesPullingTime="2.823374624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:07.768347844 +0000 UTC m=+1252.805452778" watchObservedRunningTime="2026-11-02 14:18:07.775842971 +0000 UTC m=+1252.812947902" Nov 02 14:18:08.108128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:08.108014 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" podStartSLOduration=2.221368207 podStartE2EDuration="5.10799112s" totalImagesPullingTime="2.886622913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:07.823548632 +0000 UTC m=+1252.860653565" watchObservedRunningTime="2026-11-02 14:18:08.10799112 +0000 UTC m=+1253.145096052" Nov 02 14:18:08.124526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:08.124500 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z"] Nov 02 14:18:08.159458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:08.148078 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3e2927b-dhh6z"] Nov 02 14:18:09.113801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:09.112180 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3366b2-3851-4a59-b1e7-9af8b7c70602" path="/var/lib/kubelet/pods/1c3366b2-3851-4a59-b1e7-9af8b7c70602/volumes" Nov 02 14:18:10.038528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:10.038497 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6"] Nov 02 14:18:10.051077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:10.051052 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb68d371-65kh6"] Nov 02 14:18:11.075903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:11.075870 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2339e3-6752-4dc0-ab8f-144f260b3a1a" path="/var/lib/kubelet/pods/be2339e3-6752-4dc0-ab8f-144f260b3a1a/volumes" Nov 02 14:18:12.591496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.591464 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4"] Nov 02 14:18:12.597107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.597086 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.617086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.617066 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4"] Nov 02 14:18:12.625307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.625287 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:18:12.625437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.625326 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:18:12.695614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.695589 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4h5s\" (UniqueName: \"kubernetes.io/projected/25f65433-b54b-4f98-bcd7-6e3bd65d624a-kube-api-access-w4h5s\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.695741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.695626 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.695741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.695711 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.796571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.796548 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4h5s\" (UniqueName: \"kubernetes.io/projected/25f65433-b54b-4f98-bcd7-6e3bd65d624a-kube-api-access-w4h5s\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.796688 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.796585 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.796688 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.796641 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.797346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.797328 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.798867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.798842 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.821477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.821454 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4h5s\" (UniqueName: \"kubernetes.io/projected/25f65433-b54b-4f98-bcd7-6e3bd65d624a-kube-api-access-w4h5s\") pod \"maas-api-revoke-sub-5fd44703-lb5w4\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:12.911023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:12.910999 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:13.230098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:13.230071 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4"] Nov 02 14:18:13.230098 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:13.230084 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f65433_b54b_4f98_bcd7_6e3bd65d624a.slice/crio-9311679d21782718e88dd6c88468a55cafad0050723dfb950bdcc8e73fede8c4 WatchSource:0}: Error finding container 9311679d21782718e88dd6c88468a55cafad0050723dfb950bdcc8e73fede8c4: Status 404 returned error can't find the container with id 9311679d21782718e88dd6c88468a55cafad0050723dfb950bdcc8e73fede8c4 Nov 02 14:18:13.780636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:13.780556 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" event={"ID":"25f65433-b54b-4f98-bcd7-6e3bd65d624a","Type":"ContainerStarted","Data":"11775779dcfb3e14cdd7c1df00af494358e3b1d7f62a238b8af6398e03c6ee47"} Nov 02 14:18:13.780636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:13.780607 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" event={"ID":"25f65433-b54b-4f98-bcd7-6e3bd65d624a","Type":"ContainerStarted","Data":"9311679d21782718e88dd6c88468a55cafad0050723dfb950bdcc8e73fede8c4"} Nov 02 14:18:13.826239 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:13.826193 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" podStartSLOduration=1.826180113 podStartE2EDuration="1.826180113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:13.822854112 +0000 UTC m=+1258.859959044" watchObservedRunningTime="2026-11-02 14:18:13.826180113 +0000 UTC m=+1258.863285083" Nov 02 14:18:14.032862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:14.032789 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz"] Nov 02 14:18:14.037334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:14.037311 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521ff25e-p79wz"] Nov 02 14:18:14.789583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:14.789551 4431 generic.go:356] "Generic (PLEG): container finished" podID="25f65433-b54b-4f98-bcd7-6e3bd65d624a" containerID="11775779dcfb3e14cdd7c1df00af494358e3b1d7f62a238b8af6398e03c6ee47" exitCode=0 Nov 02 14:18:14.789979 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:14.789638 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" event={"ID":"25f65433-b54b-4f98-bcd7-6e3bd65d624a","Type":"ContainerDied","Data":"11775779dcfb3e14cdd7c1df00af494358e3b1d7f62a238b8af6398e03c6ee47"} Nov 02 14:18:15.077633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.077554 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f766458-1e64-452f-84f6-e2ff95fdd36a" path="/var/lib/kubelet/pods/1f766458-1e64-452f-84f6-e2ff95fdd36a/volumes" Nov 02 14:18:15.623491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.623460 4431 scope.go:122] "RemoveContainer" containerID="726cfc75679367eb55e413331caff34e4ecc1a7327b0dfcb2b785ee1d99254eb" Nov 02 14:18:15.633474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.633457 4431 scope.go:122] "RemoveContainer" containerID="b784deef4fc89c82da6228b36e75f082090a7279bdf4ece06849ce709fa0d5bb" Nov 02 14:18:15.643148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.643130 4431 scope.go:122] "RemoveContainer" containerID="3374394f037ebb5becb59dadaffe31a63d3264302fa850d0ada215853e430bf6" Nov 02 14:18:15.651888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.651861 4431 scope.go:122] "RemoveContainer" containerID="dc44ae52caa41fc3ca8b3bcb7c716a02807f587acdd02dfb135bf775ea48b978" Nov 02 14:18:15.660285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.660270 4431 scope.go:122] "RemoveContainer" containerID="6047d2c4563145c2e96d2ec5c7c97bdc74c28aad2116e677ee5cdd55ce8e180d" Nov 02 14:18:15.668627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.668607 4431 scope.go:122] "RemoveContainer" containerID="d091fd92a7257f748a1dbb6ab27eba090fb17b88803d5225ce5c83533e83b971" Nov 02 14:18:15.677231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.677214 4431 scope.go:122] "RemoveContainer" containerID="55186ef8e940674e52668e5f016eec09d005c5d418bc610ca53ae9f7c24945db" Nov 02 14:18:15.685866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.685849 4431 scope.go:122] "RemoveContainer" containerID="641bf69f215ae0ee071d1c3f0d50bebfaafbaaadea1b88e62103d2293c518435" Nov 02 14:18:15.694034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.694018 4431 scope.go:122] "RemoveContainer" containerID="c25c5e858606a29ffef332d782b0fce4e4665029270b8134b4e385254c22fac3" Nov 02 14:18:15.702141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.702123 4431 scope.go:122] "RemoveContainer" containerID="a62e510b161590e7c5a81617ab231f29ce63ad69856ab796db5b8d6b41649373" Nov 02 14:18:15.948020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:15.947998 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:16.036546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.036515 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-xks-ca\") pod \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " Nov 02 14:18:16.036692 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.036614 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f65433-b54b-4f98-bcd7-6e3bd65d624a-kube-api-access-w4h5s\" (UniqueName: \"kubernetes.io/projected/25f65433-b54b-4f98-bcd7-6e3bd65d624a-kube-api-access-w4h5s\") pod \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " Nov 02 14:18:16.036834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.036818 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-openshift-ca\") pod \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\" (UID: \"25f65433-b54b-4f98-bcd7-6e3bd65d624a\") " Nov 02 14:18:16.037084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.037062 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-openshift-ca" pod "25f65433-b54b-4f98-bcd7-6e3bd65d624a" (UID: "25f65433-b54b-4f98-bcd7-6e3bd65d624a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:18:16.037233 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.037216 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:16.038506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.038487 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-xks-ca" pod "25f65433-b54b-4f98-bcd7-6e3bd65d624a" (UID: "25f65433-b54b-4f98-bcd7-6e3bd65d624a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:18:16.038559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.038547 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f65433-b54b-4f98-bcd7-6e3bd65d624a-kube-api-access-w4h5s" pod "25f65433-b54b-4f98-bcd7-6e3bd65d624a" (UID: "25f65433-b54b-4f98-bcd7-6e3bd65d624a"). InnerVolumeSpecName "kube-api-access-w4h5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:18:16.138592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.138564 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f65433-b54b-4f98-bcd7-6e3bd65d624a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:16.138592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.138587 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4h5s\" (UniqueName: \"kubernetes.io/projected/25f65433-b54b-4f98-bcd7-6e3bd65d624a-kube-api-access-w4h5s\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:16.811101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.811071 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" Nov 02 14:18:16.811295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.811105 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4" event={"ID":"25f65433-b54b-4f98-bcd7-6e3bd65d624a","Type":"ContainerDied","Data":"9311679d21782718e88dd6c88468a55cafad0050723dfb950bdcc8e73fede8c4"} Nov 02 14:18:16.811295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:16.811152 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9311679d21782718e88dd6c88468a55cafad0050723dfb950bdcc8e73fede8c4" Nov 02 14:18:18.744771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:18.744738 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:18:19.753635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:19.753607 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:18:20.684742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.684703 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294"] Nov 02 14:18:20.685367 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.685346 4431 cpu_manager.go:414] "containerMap: removing container" podUID="25f65433-b54b-4f98-bcd7-6e3bd65d624a" containerName="revoke-keys" Nov 02 14:18:20.685476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.685368 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f65433-b54b-4f98-bcd7-6e3bd65d624a" containerName="revoke-keys" Nov 02 14:18:20.685552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.685539 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f65433-b54b-4f98-bcd7-6e3bd65d624a" containerName="revoke-keys" Nov 02 14:18:20.689246 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.689224 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.708838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.708808 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294"] Nov 02 14:18:20.717328 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.717302 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-plugins-e2e-extmod-b-12b8da\"" Nov 02 14:18:20.717513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.717320 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"routing-overlay\"" Nov 02 14:18:20.717629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.717344 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"kube-root-ca.crt\"" Nov 02 14:18:20.717629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.717460 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"openshift-service-ca.crt\"" Nov 02 14:18:20.724765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.724747 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-047606d198bb\"" Nov 02 14:18:20.724959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.724942 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-e2e--3ffe20a0-dockercfg-kwvdn\"" Nov 02 14:18:20.795109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.795075 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.795469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.795140 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.795469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.795244 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-477lm\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-kube-api-access-477lm\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.795469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.795274 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.897165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.897130 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.897471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.897451 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.898227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.897568 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-477lm\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-kube-api-access-477lm\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.898227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.897600 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.899124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.899094 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.900183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.899764 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.918420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.918390 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:20.931161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:20.931125 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-477lm\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-kube-api-access-477lm\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:21.003189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:21.003121 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:21.284574 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:18:21.284489 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:18:21.312488 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:21.312454 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0334c06_78a9_4b73_9f96_3cfa5256a860.slice/crio-9c3318d38f60288b20b6f80301db8b88c049299d7e87d446203a9bb0243b87a8 WatchSource:0}: Error finding container 9c3318d38f60288b20b6f80301db8b88c049299d7e87d446203a9bb0243b87a8: Status 404 returned error can't find the container with id 9c3318d38f60288b20b6f80301db8b88c049299d7e87d446203a9bb0243b87a8 Nov 02 14:18:21.314078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:21.314049 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294"] Nov 02 14:18:21.887165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:21.887125 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" event={"ID":"d0334c06-78a9-4b73-9f96-3cfa5256a860","Type":"ContainerStarted","Data":"5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25"} Nov 02 14:18:21.887559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:21.887185 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" event={"ID":"d0334c06-78a9-4b73-9f96-3cfa5256a860","Type":"ContainerStarted","Data":"9c3318d38f60288b20b6f80301db8b88c049299d7e87d446203a9bb0243b87a8"} Nov 02 14:18:21.887559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:21.887213 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:21.932589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:21.932543 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" podStartSLOduration=1.932529967 podStartE2EDuration="1.932529967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:21.930306661 +0000 UTC m=+1266.967411593" watchObservedRunningTime="2026-11-02 14:18:21.932529967 +0000 UTC m=+1266.969634899" Nov 02 14:18:22.034518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.034484 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq"] Nov 02 14:18:22.037970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.037950 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01f49fb-492tq"] Nov 02 14:18:22.445057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.445026 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq"] Nov 02 14:18:22.449302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.449275 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.479825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.478462 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-598f75ee016d\"" Nov 02 14:18:22.479825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.478594 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-e2e--d8f5123f-dockercfg-qlpzh\"" Nov 02 14:18:22.479825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.478647 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ba664\"" Nov 02 14:18:22.479825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.478661 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"openshift-service-ca.crt\"" Nov 02 14:18:22.479825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.478760 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"kube-root-ca.crt\"" Nov 02 14:18:22.479825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.478855 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"routing-overlay\"" Nov 02 14:18:22.479825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.479332 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq"] Nov 02 14:18:22.516641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.516612 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lf4t\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-kube-api-access-4lf4t\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.516790 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.516748 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.516929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.516914 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.516968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.516942 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.618128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.618091 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.618332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.618133 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.618332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.618159 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lf4t\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-kube-api-access-4lf4t\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.618332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.618252 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.618985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.618957 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.619077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.619013 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.620669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.620643 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.644754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.644725 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lf4t\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-kube-api-access-4lf4t\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.764168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.764086 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:22.941735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.941707 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8"] Nov 02 14:18:22.947269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.947248 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:22.963678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.963654 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:18:22.963829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.963813 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:18:22.966409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:22.966391 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8"] Nov 02 14:18:23.024105 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.024033 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.024105 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.024061 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxbd\" (UniqueName: \"kubernetes.io/projected/02cf3032-cebe-423b-8212-5bc7512c3462-kube-api-access-8xxbd\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.024361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.024197 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.069658 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:23.069630 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode068bff5_9b0b_48c3_8d6b_54ee3c117ffa.slice/crio-f5c10690d82634423a1dc0412c847039699ed43ca5d5accd732ac4eeb5b4eac7 WatchSource:0}: Error finding container f5c10690d82634423a1dc0412c847039699ed43ca5d5accd732ac4eeb5b4eac7: Status 404 returned error can't find the container with id f5c10690d82634423a1dc0412c847039699ed43ca5d5accd732ac4eeb5b4eac7 Nov 02 14:18:23.083542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.083515 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48589894-4950-4746-8581-a3f196d7193e" path="/var/lib/kubelet/pods/48589894-4950-4746-8581-a3f196d7193e/volumes" Nov 02 14:18:23.083862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.083840 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq"] Nov 02 14:18:23.125809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.125770 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.125927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.125829 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxbd\" (UniqueName: \"kubernetes.io/projected/02cf3032-cebe-423b-8212-5bc7512c3462-kube-api-access-8xxbd\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.125927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.125886 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.126365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.126335 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.128058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.128038 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.154591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.154567 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxbd\" (UniqueName: \"kubernetes.io/projected/02cf3032-cebe-423b-8212-5bc7512c3462-kube-api-access-8xxbd\") pod \"maas-api-revoke-sub-12879879-47mg8\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.189179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.189147 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5"] Nov 02 14:18:23.193938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.193913 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.218328 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.218296 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5"] Nov 02 14:18:23.226814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.226772 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.227003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.226831 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.227003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.226858 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.227003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.226886 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw8wz\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-kube-api-access-mw8wz\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.227209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.227070 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f2d68a-4f50-4fec-afed-5a20268f857c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.227209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.227100 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.227209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.227136 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.227379 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.227217 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.227379 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.227255 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.261522 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.261493 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:23.329877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.328661 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f2d68a-4f50-4fec-afed-5a20268f857c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.330000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.329912 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.330000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.329966 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.330418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.329838 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f2d68a-4f50-4fec-afed-5a20268f857c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.330534 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.330582 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.330656 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.330692 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.330721 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.330751 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw8wz\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-kube-api-access-mw8wz\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.330849 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.331210 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.331447 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.331968 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.333905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.333647 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.336926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.336904 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.365616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.365589 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw8wz\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-kube-api-access-mw8wz\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.366913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.366886 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-cknl5\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.510931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.510898 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:23.581756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.578996 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8"] Nov 02 14:18:23.584703 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:23.584679 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cf3032_cebe_423b_8212_5bc7512c3462.slice/crio-7a1b29d59e802751a038e86d18811856a4b68cf6d28ee7b15c6bd42b3588276b WatchSource:0}: Error finding container 7a1b29d59e802751a038e86d18811856a4b68cf6d28ee7b15c6bd42b3588276b: Status 404 returned error can't find the container with id 7a1b29d59e802751a038e86d18811856a4b68cf6d28ee7b15c6bd42b3588276b Nov 02 14:18:23.821656 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:23.821629 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f2d68a_4f50_4fec_afed_5a20268f857c.slice/crio-3ac8d775bb3675d86c06b3da8c66869307f599c0dcca84a278e63df2e50833ea WatchSource:0}: Error finding container 3ac8d775bb3675d86c06b3da8c66869307f599c0dcca84a278e63df2e50833ea: Status 404 returned error can't find the container with id 3ac8d775bb3675d86c06b3da8c66869307f599c0dcca84a278e63df2e50833ea Nov 02 14:18:23.821828 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.821806 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5"] Nov 02 14:18:23.824011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.823981 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:18:23.824075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.824044 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:18:23.824119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.824074 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:18:23.903666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.903636 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" event={"ID":"09f2d68a-4f50-4fec-afed-5a20268f857c","Type":"ContainerStarted","Data":"01f1a807ee3f63eacc4ff59eac55f49a6cec4662c9d8106df8d6f194d0f4ff37"} Nov 02 14:18:23.903809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.903691 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" event={"ID":"09f2d68a-4f50-4fec-afed-5a20268f857c","Type":"ContainerStarted","Data":"3ac8d775bb3675d86c06b3da8c66869307f599c0dcca84a278e63df2e50833ea"} Nov 02 14:18:23.905053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.905020 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" event={"ID":"02cf3032-cebe-423b-8212-5bc7512c3462","Type":"ContainerStarted","Data":"7a1b29d59e802751a038e86d18811856a4b68cf6d28ee7b15c6bd42b3588276b"} Nov 02 14:18:23.906876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.906857 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" event={"ID":"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa","Type":"ContainerStarted","Data":"933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6"} Nov 02 14:18:23.906969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.906893 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" event={"ID":"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa","Type":"ContainerStarted","Data":"f5c10690d82634423a1dc0412c847039699ed43ca5d5accd732ac4eeb5b4eac7"} Nov 02 14:18:23.906969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.906940 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:23.955835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:23.955770 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podStartSLOduration=0.955757893 podStartE2EDuration="955.757893ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:23.950444931 +0000 UTC m=+1268.987549879" watchObservedRunningTime="2026-11-02 14:18:23.955757893 +0000 UTC m=+1268.992862825" Nov 02 14:18:24.007978 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:24.007932 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" podStartSLOduration=2.007918515 podStartE2EDuration="2.007918515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:24.003204558 +0000 UTC m=+1269.040309490" watchObservedRunningTime="2026-11-02 14:18:24.007918515 +0000 UTC m=+1269.045023465" Nov 02 14:18:24.511124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:24.511086 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:24.512521 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:24.512493 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:18:24.512634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:24.512541 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:18:24.915279 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:24.915247 4431 generic.go:356] "Generic (PLEG): container finished" podID="02cf3032-cebe-423b-8212-5bc7512c3462" containerID="bc65c4c8b5056e10159bc322ea3cada2d3952f48c40413483ec20e866ad2e313" exitCode=0 Nov 02 14:18:24.915486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:24.915337 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" event={"ID":"02cf3032-cebe-423b-8212-5bc7512c3462","Type":"ContainerDied","Data":"bc65c4c8b5056e10159bc322ea3cada2d3952f48c40413483ec20e866ad2e313"} Nov 02 14:18:25.511787 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:25.511747 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:18:25.512189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:25.511807 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:18:26.125703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.125681 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:26.166111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.166086 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cf3032-cebe-423b-8212-5bc7512c3462-kube-api-access-8xxbd\" (UniqueName: \"kubernetes.io/projected/02cf3032-cebe-423b-8212-5bc7512c3462-kube-api-access-8xxbd\") pod \"02cf3032-cebe-423b-8212-5bc7512c3462\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " Nov 02 14:18:26.166210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.166123 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-xks-ca\") pod \"02cf3032-cebe-423b-8212-5bc7512c3462\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " Nov 02 14:18:26.166210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.166205 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-openshift-ca\") pod \"02cf3032-cebe-423b-8212-5bc7512c3462\" (UID: \"02cf3032-cebe-423b-8212-5bc7512c3462\") " Nov 02 14:18:26.166619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.166592 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-openshift-ca" pod "02cf3032-cebe-423b-8212-5bc7512c3462" (UID: "02cf3032-cebe-423b-8212-5bc7512c3462"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:18:26.168369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.168345 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cf3032-cebe-423b-8212-5bc7512c3462-kube-api-access-8xxbd" pod "02cf3032-cebe-423b-8212-5bc7512c3462" (UID: "02cf3032-cebe-423b-8212-5bc7512c3462"). InnerVolumeSpecName "kube-api-access-8xxbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:18:26.168516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.168490 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-xks-ca" pod "02cf3032-cebe-423b-8212-5bc7512c3462" (UID: "02cf3032-cebe-423b-8212-5bc7512c3462"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:18:26.267652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.267620 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xxbd\" (UniqueName: \"kubernetes.io/projected/02cf3032-cebe-423b-8212-5bc7512c3462-kube-api-access-8xxbd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:26.267652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.267649 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:26.267652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.267659 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02cf3032-cebe-423b-8212-5bc7512c3462-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:26.512215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.512130 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:18:26.512215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.512197 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:18:26.939465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.939437 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" Nov 02 14:18:26.939609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.939473 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8" event={"ID":"02cf3032-cebe-423b-8212-5bc7512c3462","Type":"ContainerDied","Data":"7a1b29d59e802751a038e86d18811856a4b68cf6d28ee7b15c6bd42b3588276b"} Nov 02 14:18:26.939609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:26.939545 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a1b29d59e802751a038e86d18811856a4b68cf6d28ee7b15c6bd42b3588276b" Nov 02 14:18:27.511672 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:27.511638 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:18:27.511892 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:27.511692 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:18:28.511946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:28.511910 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:18:28.512333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:28.511962 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:18:29.511676 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:29.511646 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:18:29.511888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:29.511692 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:18:30.512120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:30.512081 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-cknl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 14:18:30.512514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:30.512155 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 14:18:32.245466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:32.245437 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:32.245949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:32.245929 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:32.246638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:32.246619 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:18:32.897159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:32.897132 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:18:34.918882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:34.918843 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:18:39.437740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.437701 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs"] Nov 02 14:18:39.438586 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.438538 4431 cpu_manager.go:414] "containerMap: removing container" podUID="02cf3032-cebe-423b-8212-5bc7512c3462" containerName="revoke-keys" Nov 02 14:18:39.438586 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.438583 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cf3032-cebe-423b-8212-5bc7512c3462" containerName="revoke-keys" Nov 02 14:18:39.438769 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.438756 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cf3032-cebe-423b-8212-5bc7512c3462" containerName="revoke-keys" Nov 02 14:18:39.449000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.448927 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.462982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.462955 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs"] Nov 02 14:18:39.470896 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.470876 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:18:39.471022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.470944 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:18:39.491299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.491273 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcmh\" (UniqueName: \"kubernetes.io/projected/6a3656a8-c36f-496e-8898-49a6ff62837f-kube-api-access-wzcmh\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.491418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.491318 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.491488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.491469 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.593110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.593084 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.593344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.593155 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcmh\" (UniqueName: \"kubernetes.io/projected/6a3656a8-c36f-496e-8898-49a6ff62837f-kube-api-access-wzcmh\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.593344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.593175 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.593811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.593761 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.595492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.595475 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.617602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.617579 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcmh\" (UniqueName: \"kubernetes.io/projected/6a3656a8-c36f-496e-8898-49a6ff62837f-kube-api-access-wzcmh\") pod \"maas-api-revoke-sub-0d89100b-jqdjs\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:39.765621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:39.765541 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:40.077021 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:40.076989 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3656a8_c36f_496e_8898_49a6ff62837f.slice/crio-8b7c207188de3d44cc026ba87296dc2314d8dfc8a03cca9411a263664a51ae90 WatchSource:0}: Error finding container 8b7c207188de3d44cc026ba87296dc2314d8dfc8a03cca9411a263664a51ae90: Status 404 returned error can't find the container with id 8b7c207188de3d44cc026ba87296dc2314d8dfc8a03cca9411a263664a51ae90 Nov 02 14:18:40.082465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:40.082444 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs"] Nov 02 14:18:41.079462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:41.079426 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" event={"ID":"6a3656a8-c36f-496e-8898-49a6ff62837f","Type":"ContainerStarted","Data":"edf7f13389113ed83d0a43574cc0dc3bbc5c5505ae5ecca223eb32b130d19371"} Nov 02 14:18:41.079858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:41.079486 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" event={"ID":"6a3656a8-c36f-496e-8898-49a6ff62837f","Type":"ContainerStarted","Data":"8b7c207188de3d44cc026ba87296dc2314d8dfc8a03cca9411a263664a51ae90"} Nov 02 14:18:41.124957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:41.124912 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" podStartSLOduration=2.124898426 podStartE2EDuration="2.124898426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:41.12244393 +0000 UTC m=+1286.159548862" watchObservedRunningTime="2026-11-02 14:18:41.124898426 +0000 UTC m=+1286.162003352" Nov 02 14:18:42.088707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:42.088671 4431 generic.go:356] "Generic (PLEG): container finished" podID="6a3656a8-c36f-496e-8898-49a6ff62837f" containerID="edf7f13389113ed83d0a43574cc0dc3bbc5c5505ae5ecca223eb32b130d19371" exitCode=0 Nov 02 14:18:42.089155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:42.088772 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" event={"ID":"6a3656a8-c36f-496e-8898-49a6ff62837f","Type":"ContainerDied","Data":"edf7f13389113ed83d0a43574cc0dc3bbc5c5505ae5ecca223eb32b130d19371"} Nov 02 14:18:43.308895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.308871 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:43.340207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.340180 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-openshift-ca\") pod \"6a3656a8-c36f-496e-8898-49a6ff62837f\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " Nov 02 14:18:43.340372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.340314 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3656a8-c36f-496e-8898-49a6ff62837f-kube-api-access-wzcmh\" (UniqueName: \"kubernetes.io/projected/6a3656a8-c36f-496e-8898-49a6ff62837f-kube-api-access-wzcmh\") pod \"6a3656a8-c36f-496e-8898-49a6ff62837f\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " Nov 02 14:18:43.340420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.340385 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-xks-ca\") pod \"6a3656a8-c36f-496e-8898-49a6ff62837f\" (UID: \"6a3656a8-c36f-496e-8898-49a6ff62837f\") " Nov 02 14:18:43.340551 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.340526 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-openshift-ca" pod "6a3656a8-c36f-496e-8898-49a6ff62837f" (UID: "6a3656a8-c36f-496e-8898-49a6ff62837f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:18:43.340865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.340847 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:43.342352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.342332 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-xks-ca" pod "6a3656a8-c36f-496e-8898-49a6ff62837f" (UID: "6a3656a8-c36f-496e-8898-49a6ff62837f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:18:43.342497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.342479 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3656a8-c36f-496e-8898-49a6ff62837f-kube-api-access-wzcmh" pod "6a3656a8-c36f-496e-8898-49a6ff62837f" (UID: "6a3656a8-c36f-496e-8898-49a6ff62837f"). InnerVolumeSpecName "kube-api-access-wzcmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:18:43.442480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.442447 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzcmh\" (UniqueName: \"kubernetes.io/projected/6a3656a8-c36f-496e-8898-49a6ff62837f-kube-api-access-wzcmh\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:43.442480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:43.442480 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a3656a8-c36f-496e-8898-49a6ff62837f-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:44.118891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:44.118862 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" Nov 02 14:18:44.119063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:44.118899 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs" event={"ID":"6a3656a8-c36f-496e-8898-49a6ff62837f","Type":"ContainerDied","Data":"8b7c207188de3d44cc026ba87296dc2314d8dfc8a03cca9411a263664a51ae90"} Nov 02 14:18:44.119063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:44.118936 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7c207188de3d44cc026ba87296dc2314d8dfc8a03cca9411a263664a51ae90" Nov 02 14:18:49.757333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.757284 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44"] Nov 02 14:18:49.758251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.758225 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6a3656a8-c36f-496e-8898-49a6ff62837f" containerName="revoke-keys" Nov 02 14:18:49.758365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.758253 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3656a8-c36f-496e-8898-49a6ff62837f" containerName="revoke-keys" Nov 02 14:18:49.758477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.758461 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3656a8-c36f-496e-8898-49a6ff62837f" containerName="revoke-keys" Nov 02 14:18:49.762493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.762469 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.783115 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.783091 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44"] Nov 02 14:18:49.792099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.792075 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e1c2\"/\"openshift-service-ca.crt\"" Nov 02 14:18:49.792099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.792094 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e1c2\"/\"kube-root-ca.crt\"" Nov 02 14:18:49.792290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.792236 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e1c2\"/\"default-dockercfg-8rtgf\"" Nov 02 14:18:49.792347 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.792301 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e1c2\"/\"auth-test-model-f1e1c2-kserve-self-signed-certs\"" Nov 02 14:18:49.832907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.832871 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdl9\" (UniqueName: \"kubernetes.io/projected/04f711d0-98ec-4d7f-936b-b3d223cba1b2-kube-api-access-rxdl9\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.833069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.832912 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-dshm\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.833069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.833028 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tls-certs\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.833145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.833125 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-model-cache\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.833286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.833266 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-home\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.833324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.833303 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tmp-dir\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.934859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.934813 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-home\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.935026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.934866 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tmp-dir\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.935026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.934992 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdl9\" (UniqueName: \"kubernetes.io/projected/04f711d0-98ec-4d7f-936b-b3d223cba1b2-kube-api-access-rxdl9\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.935137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.935025 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-dshm\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.935137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.935094 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tls-certs\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.935238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.935151 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-model-cache\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.935331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.935303 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-home\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.935404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.935326 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tmp-dir\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.935528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.935507 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-model-cache\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.937382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.937356 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-dshm\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.937893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.937870 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tls-certs\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:49.959813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:49.959767 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdl9\" (UniqueName: \"kubernetes.io/projected/04f711d0-98ec-4d7f-936b-b3d223cba1b2-kube-api-access-rxdl9\") pod \"auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:50.077204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:50.077123 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:50.435321 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:50.435279 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04f711d0_98ec_4d7f_936b_b3d223cba1b2.slice/crio-f310e642c2abe15ada6399818e8673caae5fe0398bbd2431fcda3df317ba0603 WatchSource:0}: Error finding container f310e642c2abe15ada6399818e8673caae5fe0398bbd2431fcda3df317ba0603: Status 404 returned error can't find the container with id f310e642c2abe15ada6399818e8673caae5fe0398bbd2431fcda3df317ba0603 Nov 02 14:18:50.437252 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:50.437227 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44"] Nov 02 14:18:51.189666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:51.189617 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" event={"ID":"04f711d0-98ec-4d7f-936b-b3d223cba1b2","Type":"ContainerStarted","Data":"a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2"} Nov 02 14:18:51.189666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:51.189666 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" event={"ID":"04f711d0-98ec-4d7f-936b-b3d223cba1b2","Type":"ContainerStarted","Data":"f310e642c2abe15ada6399818e8673caae5fe0398bbd2431fcda3df317ba0603"} Nov 02 14:18:51.238603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:51.238531 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" podStartSLOduration=2.238513928 podStartE2EDuration="2.238513928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:51.234007617 +0000 UTC m=+1296.271112549" watchObservedRunningTime="2026-11-02 14:18:51.238513928 +0000 UTC m=+1296.275618861" Nov 02 14:18:51.283538 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:18:51.283502 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:18:52.077698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:52.077669 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:18:54.027455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.027424 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb"] Nov 02 14:18:54.032693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.032674 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.053808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.053770 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb"] Nov 02 14:18:54.060656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.060630 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:18:54.060771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.060755 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:18:54.096990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.096964 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.097080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.096999 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8vd\" (UniqueName: \"kubernetes.io/projected/27ad3e28-230b-4cfe-a186-d8afcf0e1320-kube-api-access-dh8vd\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.097134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.097116 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.198077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.198049 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.198224 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.198210 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.198274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.198238 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8vd\" (UniqueName: \"kubernetes.io/projected/27ad3e28-230b-4cfe-a186-d8afcf0e1320-kube-api-access-dh8vd\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.198903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.198884 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.200395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.200377 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.223028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.222997 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8vd\" (UniqueName: \"kubernetes.io/projected/27ad3e28-230b-4cfe-a186-d8afcf0e1320-kube-api-access-dh8vd\") pod \"maas-api-revoke-sub-0b8ef107-n56pb\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.347104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.347039 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:54.656130 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:54.656099 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ad3e28_230b_4cfe_a186_d8afcf0e1320.slice/crio-e91b4ad6d14b473bfe75ba335b2e14545b7c695adb6fe7d576a72672c42cbf46 WatchSource:0}: Error finding container e91b4ad6d14b473bfe75ba335b2e14545b7c695adb6fe7d576a72672c42cbf46: Status 404 returned error can't find the container with id e91b4ad6d14b473bfe75ba335b2e14545b7c695adb6fe7d576a72672c42cbf46 Nov 02 14:18:54.656303 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:54.656280 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb"] Nov 02 14:18:55.220003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.219966 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" event={"ID":"27ad3e28-230b-4cfe-a186-d8afcf0e1320","Type":"ContainerStarted","Data":"e1ef917a53d451a6b822a6bcfb24aa52db08dead5f029d29d2fbf660298f3edd"} Nov 02 14:18:55.220369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.220027 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" event={"ID":"27ad3e28-230b-4cfe-a186-d8afcf0e1320","Type":"ContainerStarted","Data":"e91b4ad6d14b473bfe75ba335b2e14545b7c695adb6fe7d576a72672c42cbf46"} Nov 02 14:18:55.265111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.265008 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" podStartSLOduration=2.264989594 podStartE2EDuration="2.264989594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:55.262393787 +0000 UTC m=+1300.299498720" watchObservedRunningTime="2026-11-02 14:18:55.264989594 +0000 UTC m=+1300.302094526" Nov 02 14:18:55.626370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.626337 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw"] Nov 02 14:18:55.631004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.630975 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.650953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.650921 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw"] Nov 02 14:18:55.717220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.717187 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.717363 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.717239 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cgp\" (UniqueName: \"kubernetes.io/projected/3faa43eb-60a8-4730-aee4-817f2feff6a4-kube-api-access-h5cgp\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.717415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.717372 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.818607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.818577 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.818750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.818618 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cgp\" (UniqueName: \"kubernetes.io/projected/3faa43eb-60a8-4730-aee4-817f2feff6a4-kube-api-access-h5cgp\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.818750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.818678 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.819347 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.819316 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.821049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.821025 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.843203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.843150 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cgp\" (UniqueName: \"kubernetes.io/projected/3faa43eb-60a8-4730-aee4-817f2feff6a4-kube-api-access-h5cgp\") pod \"maas-api-revoke-sub-e6eeee79-29wvw\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:55.959756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:55.959729 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:56.229245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:56.229213 4431 generic.go:356] "Generic (PLEG): container finished" podID="27ad3e28-230b-4cfe-a186-d8afcf0e1320" containerID="e1ef917a53d451a6b822a6bcfb24aa52db08dead5f029d29d2fbf660298f3edd" exitCode=0 Nov 02 14:18:56.229610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:56.229289 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" event={"ID":"27ad3e28-230b-4cfe-a186-d8afcf0e1320","Type":"ContainerDied","Data":"e1ef917a53d451a6b822a6bcfb24aa52db08dead5f029d29d2fbf660298f3edd"} Nov 02 14:18:56.270259 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:18:56.270233 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faa43eb_60a8_4730_aee4_817f2feff6a4.slice/crio-d4870ae8ce4c4040e226de2b473258f1705efeba901c17a684dc370504259b31 WatchSource:0}: Error finding container d4870ae8ce4c4040e226de2b473258f1705efeba901c17a684dc370504259b31: Status 404 returned error can't find the container with id d4870ae8ce4c4040e226de2b473258f1705efeba901c17a684dc370504259b31 Nov 02 14:18:56.270377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:56.270278 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw"] Nov 02 14:18:57.238555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.238509 4431 generic.go:356] "Generic (PLEG): container finished" podID="3faa43eb-60a8-4730-aee4-817f2feff6a4" containerID="555e7278dcdb92002a158f8f76c63ff6e93a16b974643486b142cfcff1afaac7" exitCode=0 Nov 02 14:18:57.239002 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.238588 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" event={"ID":"3faa43eb-60a8-4730-aee4-817f2feff6a4","Type":"ContainerDied","Data":"555e7278dcdb92002a158f8f76c63ff6e93a16b974643486b142cfcff1afaac7"} Nov 02 14:18:57.239002 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.238647 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" event={"ID":"3faa43eb-60a8-4730-aee4-817f2feff6a4","Type":"ContainerStarted","Data":"d4870ae8ce4c4040e226de2b473258f1705efeba901c17a684dc370504259b31"} Nov 02 14:18:57.396761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.396739 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:57.541603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.541522 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ad3e28-230b-4cfe-a186-d8afcf0e1320-kube-api-access-dh8vd\" (UniqueName: \"kubernetes.io/projected/27ad3e28-230b-4cfe-a186-d8afcf0e1320-kube-api-access-dh8vd\") pod \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " Nov 02 14:18:57.541750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.541700 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-xks-ca\") pod \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " Nov 02 14:18:57.541826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.541811 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-openshift-ca\") pod \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\" (UID: \"27ad3e28-230b-4cfe-a186-d8afcf0e1320\") " Nov 02 14:18:57.542162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.542135 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-openshift-ca" pod "27ad3e28-230b-4cfe-a186-d8afcf0e1320" (UID: "27ad3e28-230b-4cfe-a186-d8afcf0e1320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:18:57.542310 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.542240 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:57.543694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.543671 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ad3e28-230b-4cfe-a186-d8afcf0e1320-kube-api-access-dh8vd" pod "27ad3e28-230b-4cfe-a186-d8afcf0e1320" (UID: "27ad3e28-230b-4cfe-a186-d8afcf0e1320"). InnerVolumeSpecName "kube-api-access-dh8vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:18:57.543904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.543887 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-xks-ca" pod "27ad3e28-230b-4cfe-a186-d8afcf0e1320" (UID: "27ad3e28-230b-4cfe-a186-d8afcf0e1320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:18:57.644030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.643991 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ad3e28-230b-4cfe-a186-d8afcf0e1320-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:57.644131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:57.644034 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh8vd\" (UniqueName: \"kubernetes.io/projected/27ad3e28-230b-4cfe-a186-d8afcf0e1320-kube-api-access-dh8vd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:58.248533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.248505 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" Nov 02 14:18:58.248941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.248535 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb" event={"ID":"27ad3e28-230b-4cfe-a186-d8afcf0e1320","Type":"ContainerDied","Data":"e91b4ad6d14b473bfe75ba335b2e14545b7c695adb6fe7d576a72672c42cbf46"} Nov 02 14:18:58.248941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.248583 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e91b4ad6d14b473bfe75ba335b2e14545b7c695adb6fe7d576a72672c42cbf46" Nov 02 14:18:58.396251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.396226 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:58.554686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.554613 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faa43eb-60a8-4730-aee4-817f2feff6a4-kube-api-access-h5cgp\" (UniqueName: \"kubernetes.io/projected/3faa43eb-60a8-4730-aee4-817f2feff6a4-kube-api-access-h5cgp\") pod \"3faa43eb-60a8-4730-aee4-817f2feff6a4\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " Nov 02 14:18:58.554686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.554652 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-openshift-ca\") pod \"3faa43eb-60a8-4730-aee4-817f2feff6a4\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " Nov 02 14:18:58.554944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.554726 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-xks-ca\") pod \"3faa43eb-60a8-4730-aee4-817f2feff6a4\" (UID: \"3faa43eb-60a8-4730-aee4-817f2feff6a4\") " Nov 02 14:18:58.555061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.555040 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-openshift-ca" pod "3faa43eb-60a8-4730-aee4-817f2feff6a4" (UID: "3faa43eb-60a8-4730-aee4-817f2feff6a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:18:58.555220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.555205 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:58.556764 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.556738 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faa43eb-60a8-4730-aee4-817f2feff6a4-kube-api-access-h5cgp" pod "3faa43eb-60a8-4730-aee4-817f2feff6a4" (UID: "3faa43eb-60a8-4730-aee4-817f2feff6a4"). InnerVolumeSpecName "kube-api-access-h5cgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:18:58.556930 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.556741 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-xks-ca" pod "3faa43eb-60a8-4730-aee4-817f2feff6a4" (UID: "3faa43eb-60a8-4730-aee4-817f2feff6a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:18:58.656037 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.656006 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5cgp\" (UniqueName: \"kubernetes.io/projected/3faa43eb-60a8-4730-aee4-817f2feff6a4-kube-api-access-h5cgp\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:58.656037 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:58.656034 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3faa43eb-60a8-4730-aee4-817f2feff6a4-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:18:59.257540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:59.257508 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" Nov 02 14:18:59.257540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:59.257510 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw" event={"ID":"3faa43eb-60a8-4730-aee4-817f2feff6a4","Type":"ContainerDied","Data":"d4870ae8ce4c4040e226de2b473258f1705efeba901c17a684dc370504259b31"} Nov 02 14:18:59.257965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:59.257573 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4870ae8ce4c4040e226de2b473258f1705efeba901c17a684dc370504259b31" Nov 02 14:18:59.442972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:18:59.442943 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:19:00.077986 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:00.077958 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:19:00.137328 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:00.137298 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:19:00.289622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:00.289592 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:19:07.503439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.503410 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f"] Nov 02 14:19:07.504014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.503998 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3faa43eb-60a8-4730-aee4-817f2feff6a4" containerName="revoke-keys" Nov 02 14:19:07.504070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.504014 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3faa43eb-60a8-4730-aee4-817f2feff6a4" containerName="revoke-keys" Nov 02 14:19:07.504070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.504032 4431 cpu_manager.go:414] "containerMap: removing container" podUID="27ad3e28-230b-4cfe-a186-d8afcf0e1320" containerName="revoke-keys" Nov 02 14:19:07.504070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.504038 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ad3e28-230b-4cfe-a186-d8afcf0e1320" containerName="revoke-keys" Nov 02 14:19:07.504175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.504165 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3faa43eb-60a8-4730-aee4-817f2feff6a4" containerName="revoke-keys" Nov 02 14:19:07.504210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.504180 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ad3e28-230b-4cfe-a186-d8afcf0e1320" containerName="revoke-keys" Nov 02 14:19:07.507660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.507644 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.531438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.531398 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f"] Nov 02 14:19:07.535254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.534750 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:19:07.535254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.534861 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:19:07.634701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.634665 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.634940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.634861 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59z6b\" (UniqueName: \"kubernetes.io/projected/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-kube-api-access-59z6b\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.634940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.634914 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.736154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.736124 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59z6b\" (UniqueName: \"kubernetes.io/projected/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-kube-api-access-59z6b\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.736319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.736165 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.736319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.736308 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.736890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.736858 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.738757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.738728 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.760663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.760607 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59z6b\" (UniqueName: \"kubernetes.io/projected/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-kube-api-access-59z6b\") pod \"maas-api-revoke-sub-9d4d9d31-rq57f\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:07.823053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:07.823016 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:08.135391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:08.135364 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f"] Nov 02 14:19:08.135900 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:19:08.135871 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b1c0e7_2a7e_4a1d_b9b4_9b05d1a44972.slice/crio-d8db42f59b138ba52a3859753241c52d8f8bf124d6d7ad44f0de8fac4ffe9895 WatchSource:0}: Error finding container d8db42f59b138ba52a3859753241c52d8f8bf124d6d7ad44f0de8fac4ffe9895: Status 404 returned error can't find the container with id d8db42f59b138ba52a3859753241c52d8f8bf124d6d7ad44f0de8fac4ffe9895 Nov 02 14:19:08.356516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:08.356480 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" event={"ID":"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972","Type":"ContainerStarted","Data":"d8db42f59b138ba52a3859753241c52d8f8bf124d6d7ad44f0de8fac4ffe9895"} Nov 02 14:19:09.365763 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:09.365680 4431 generic.go:356] "Generic (PLEG): container finished" podID="65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972" containerID="5250e397fb962b329817380b95f14d4fbbe63884c6ed669358afb7a04243e340" exitCode=0 Nov 02 14:19:09.366237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:09.365763 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" event={"ID":"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972","Type":"ContainerDied","Data":"5250e397fb962b329817380b95f14d4fbbe63884c6ed669358afb7a04243e340"} Nov 02 14:19:10.532151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.532125 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:10.575448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.575418 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-openshift-ca\") pod \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " Nov 02 14:19:10.575577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.575534 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-xks-ca\") pod \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " Nov 02 14:19:10.575709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.575635 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-kube-api-access-59z6b\" (UniqueName: \"kubernetes.io/projected/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-kube-api-access-59z6b\") pod \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\" (UID: \"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972\") " Nov 02 14:19:10.575870 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.575841 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-openshift-ca" pod "65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972" (UID: "65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:10.576149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.576129 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:10.577738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.577711 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-xks-ca" pod "65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972" (UID: "65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:19:10.577959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.577935 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-kube-api-access-59z6b" pod "65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972" (UID: "65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972"). InnerVolumeSpecName "kube-api-access-59z6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:10.677543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.677514 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59z6b\" (UniqueName: \"kubernetes.io/projected/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-kube-api-access-59z6b\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:10.677543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:10.677541 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:11.382146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:11.382079 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" Nov 02 14:19:11.382293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:11.382077 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f" event={"ID":"65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972","Type":"ContainerDied","Data":"d8db42f59b138ba52a3859753241c52d8f8bf124d6d7ad44f0de8fac4ffe9895"} Nov 02 14:19:11.382293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:11.382207 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8db42f59b138ba52a3859753241c52d8f8bf124d6d7ad44f0de8fac4ffe9895" Nov 02 14:19:13.033341 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:13.033314 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb"] Nov 02 14:19:13.046499 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:13.046471 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e2dffb-sq5fb"] Nov 02 14:19:13.076226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:13.076193 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7ce241-aa59-4811-81e6-d61daa007fc0" path="/var/lib/kubelet/pods/8a7ce241-aa59-4811-81e6-d61daa007fc0/volumes" Nov 02 14:19:15.800068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.800035 4431 scope.go:122] "RemoveContainer" containerID="8c85bf496df3911600f39eba8afcdeb2a7faf97c4b865b77e3a9628eed942834" Nov 02 14:19:15.812956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.812936 4431 scope.go:122] "RemoveContainer" containerID="15ace6f857a85ac96cbb71e5c6337f0ffe2b90f04fe100805718049b38681a63" Nov 02 14:19:15.824540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.824515 4431 scope.go:122] "RemoveContainer" containerID="5266c4843f63549eeb86eedd71c9f83a170c92aedf075d13ad28285a5205a7b3" Nov 02 14:19:15.834508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.834482 4431 scope.go:122] "RemoveContainer" containerID="13b641a0a836eef23ab43bff96e4cc33934376722ef47d32505cbd50af79c539" Nov 02 14:19:15.847112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.846869 4431 scope.go:122] "RemoveContainer" containerID="2963455098e3a2233495cb5195194457da6ad3836043d1fa1457f2720e6c42c1" Nov 02 14:19:15.857901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.857883 4431 scope.go:122] "RemoveContainer" containerID="45379739ae82042ba175f332a7a2726e720ec8208a622f9e8653d15e491b4a16" Nov 02 14:19:15.866924 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.866906 4431 scope.go:122] "RemoveContainer" containerID="d16bec4b9ef098e21063544b393670d9381b831b52cd83c6148d753da0f5ac26" Nov 02 14:19:15.875811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.875767 4431 scope.go:122] "RemoveContainer" containerID="8dfa06455dee55835bf32aad8a2c5151ac9d3c8476219c79d67331ac2a6fbdfd" Nov 02 14:19:15.887412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.887396 4431 scope.go:122] "RemoveContainer" containerID="766bc3c3c4ea09d314f4f7f545c5d8daa24754dd862452088f34ab36c2bb5d11" Nov 02 14:19:15.896255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.896236 4431 scope.go:122] "RemoveContainer" containerID="4e4ea777f2155e44d63e92eaee2c708971e84623e3fb7726a7aa63e76b04b233" Nov 02 14:19:15.907160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:15.907135 4431 scope.go:122] "RemoveContainer" containerID="421ee41457cf0ada493bb275b1faae2eabb7a30529bb373082b00cbb0dd1a12d" Nov 02 14:19:18.447768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.447738 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz"] Nov 02 14:19:18.448376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.448361 4431 cpu_manager.go:414] "containerMap: removing container" podUID="65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972" containerName="revoke-keys" Nov 02 14:19:18.448376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.448376 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972" containerName="revoke-keys" Nov 02 14:19:18.448516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.448507 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972" containerName="revoke-keys" Nov 02 14:19:18.453687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.453667 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.476428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.476401 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz"] Nov 02 14:19:18.483316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.483297 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:19:18.483415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.483342 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:19:18.574764 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.574736 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.574959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.574822 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4lh\" (UniqueName: \"kubernetes.io/projected/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-kube-api-access-5g4lh\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.574959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.574919 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.675974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.675941 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.676147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.676021 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4lh\" (UniqueName: \"kubernetes.io/projected/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-kube-api-access-5g4lh\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.676147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.676047 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.676743 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.676715 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.678266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.678250 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.701402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.701334 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4lh\" (UniqueName: \"kubernetes.io/projected/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-kube-api-access-5g4lh\") pod \"maas-api-revoke-sub-6addd867-8kbzz\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.767545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.767519 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:18.953139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.953061 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v"] Nov 02 14:19:18.958809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.958768 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:18.976371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.976345 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-623b4b\"/\"default-dockercfg-79j45\"" Nov 02 14:19:18.976522 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.976408 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-623b4b\"/\"auth-test-model-623b4b-kserve-self-signed-certs\"" Nov 02 14:19:18.976703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.976683 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-623b4b\"/\"kube-root-ca.crt\"" Nov 02 14:19:18.976799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.976690 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-623b4b\"/\"openshift-service-ca.crt\"" Nov 02 14:19:18.980313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:18.980292 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v"] Nov 02 14:19:19.084667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.084634 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-home\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.084837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.084677 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-tmp-dir\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.084837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.084742 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-dshm\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.084974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.084856 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3079ecc4-301f-482a-ae74-5234a2a97c6c-tls-certs\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.084974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.084883 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj6q6\" (UniqueName: \"kubernetes.io/projected/3079ecc4-301f-482a-ae74-5234a2a97c6c-kube-api-access-lj6q6\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.084974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.084918 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-model-cache\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.086798 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:19:19.086755 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcd2d3a_e64f_43fe_8e5f_ea6eba61608e.slice/crio-5998af1d73f6af628e5056c74085b6be0557f62d7878925aef5b609477c4e5ed WatchSource:0}: Error finding container 5998af1d73f6af628e5056c74085b6be0557f62d7878925aef5b609477c4e5ed: Status 404 returned error can't find the container with id 5998af1d73f6af628e5056c74085b6be0557f62d7878925aef5b609477c4e5ed Nov 02 14:19:19.087068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.087045 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz"] Nov 02 14:19:19.186629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.186595 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-home\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.186805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.186636 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-tmp-dir\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.186805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.186703 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-dshm\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.187200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.187176 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-home\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.187297 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.187286 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3079ecc4-301f-482a-ae74-5234a2a97c6c-tls-certs\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.187353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.187316 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6q6\" (UniqueName: \"kubernetes.io/projected/3079ecc4-301f-482a-ae74-5234a2a97c6c-kube-api-access-lj6q6\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.187415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.187357 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-model-cache\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.188739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.188446 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-model-cache\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.189423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.189035 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-tmp-dir\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.197800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.196026 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-dshm\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.197800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.197344 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3079ecc4-301f-482a-ae74-5234a2a97c6c-tls-certs\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.217844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.214575 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj6q6\" (UniqueName: \"kubernetes.io/projected/3079ecc4-301f-482a-ae74-5234a2a97c6c-kube-api-access-lj6q6\") pod \"auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.272959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.272891 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:19.448552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.448521 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" event={"ID":"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e","Type":"ContainerStarted","Data":"5bb5782ecc18602f6e98e6019861e44442c5b3dc13bd5fd814765326f541ca28"} Nov 02 14:19:19.448943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.448582 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" event={"ID":"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e","Type":"ContainerStarted","Data":"5998af1d73f6af628e5056c74085b6be0557f62d7878925aef5b609477c4e5ed"} Nov 02 14:19:19.495831 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.495743 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" podStartSLOduration=1.495730662 podStartE2EDuration="1.495730662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:19.495310925 +0000 UTC m=+1324.532415858" watchObservedRunningTime="2026-11-02 14:19:19.495730662 +0000 UTC m=+1324.532835594" Nov 02 14:19:19.591351 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:19:19.591322 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3079ecc4_301f_482a_ae74_5234a2a97c6c.slice/crio-322350617cd3021e3bc9f87bd6a61c76107c7a1a3cf3853f470322ba046de458 WatchSource:0}: Error finding container 322350617cd3021e3bc9f87bd6a61c76107c7a1a3cf3853f470322ba046de458: Status 404 returned error can't find the container with id 322350617cd3021e3bc9f87bd6a61c76107c7a1a3cf3853f470322ba046de458 Nov 02 14:19:19.591675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:19.591651 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v"] Nov 02 14:19:20.459473 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:20.459442 4431 generic.go:356] "Generic (PLEG): container finished" podID="6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e" containerID="5bb5782ecc18602f6e98e6019861e44442c5b3dc13bd5fd814765326f541ca28" exitCode=0 Nov 02 14:19:20.459938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:20.459527 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" event={"ID":"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e","Type":"ContainerDied","Data":"5bb5782ecc18602f6e98e6019861e44442c5b3dc13bd5fd814765326f541ca28"} Nov 02 14:19:20.461216 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:20.461193 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" event={"ID":"3079ecc4-301f-482a-ae74-5234a2a97c6c","Type":"ContainerStarted","Data":"ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7"} Nov 02 14:19:20.461216 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:20.461223 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" event={"ID":"3079ecc4-301f-482a-ae74-5234a2a97c6c","Type":"ContainerStarted","Data":"322350617cd3021e3bc9f87bd6a61c76107c7a1a3cf3853f470322ba046de458"} Nov 02 14:19:20.552090 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:20.552038 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" podStartSLOduration=2.55202521 podStartE2EDuration="2.55202521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:20.548072743 +0000 UTC m=+1325.585177680" watchObservedRunningTime="2026-11-02 14:19:20.55202521 +0000 UTC m=+1325.589130141" Nov 02 14:19:21.274042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:21.274009 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:21.285723 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:19:21.285690 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:19:22.244811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.241414 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:22.397027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.392826 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-openshift-ca\") pod \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " Nov 02 14:19:22.397336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.396978 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-openshift-ca" pod "6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e" (UID: "6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:22.403215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.397769 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-kube-api-access-5g4lh\" (UniqueName: \"kubernetes.io/projected/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-kube-api-access-5g4lh\") pod \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " Nov 02 14:19:22.403435 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.403419 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-xks-ca\") pod \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\" (UID: \"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e\") " Nov 02 14:19:22.437835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.409604 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:22.626699 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.625356 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-xks-ca" pod "6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e" (UID: "6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:19:22.667866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.651955 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:22.684738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.684710 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" Nov 02 14:19:22.685161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.685112 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz" event={"ID":"6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e","Type":"ContainerDied","Data":"5998af1d73f6af628e5056c74085b6be0557f62d7878925aef5b609477c4e5ed"} Nov 02 14:19:22.685328 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.685310 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5998af1d73f6af628e5056c74085b6be0557f62d7878925aef5b609477c4e5ed" Nov 02 14:19:22.685661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.685637 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-kube-api-access-5g4lh" pod "6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e" (UID: "6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e"). InnerVolumeSpecName "kube-api-access-5g4lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:22.757841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:22.757789 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g4lh\" (UniqueName: \"kubernetes.io/projected/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e-kube-api-access-5g4lh\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:25.064858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:25.064823 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm"] Nov 02 14:19:25.112945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:25.112917 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm"] Nov 02 14:19:25.122007 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:19:25.121975 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-e5162db5-4qgsm\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5162db5-4qgsm" Nov 02 14:19:25.177677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:25.177636 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Nov 02 14:19:25.285800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:25.284853 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:19:25.290797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:25.287499 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:19:25.314805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:25.309845 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Nov 02 14:19:25.499915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:25.499870 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:19:25.551568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:25.550847 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:19:27.077327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:27.077295 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b" path="/var/lib/kubelet/pods/8fc3fe90-0460-4eff-83ef-1fe08c2f2f3b/volumes" Nov 02 14:19:29.273202 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:29.273170 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:29.286531 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:29.286509 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:29.792543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:29.792516 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:19:33.098510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.098483 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv"] Nov 02 14:19:33.099275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.099248 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e" containerName="revoke-keys" Nov 02 14:19:33.099275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.099274 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e" containerName="revoke-keys" Nov 02 14:19:33.099482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.099432 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e" containerName="revoke-keys" Nov 02 14:19:33.104670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.104651 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.122204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.122185 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv"] Nov 02 14:19:33.130485 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.130460 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:19:33.130648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.130586 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:19:33.204923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.204893 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.205091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.205031 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.205234 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.205212 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7gv\" (UniqueName: \"kubernetes.io/projected/c851ac2e-f840-4b78-bd26-85a3653ff657-kube-api-access-5w7gv\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.306145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.306117 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.306324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.306256 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7gv\" (UniqueName: \"kubernetes.io/projected/c851ac2e-f840-4b78-bd26-85a3653ff657-kube-api-access-5w7gv\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.306398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.306360 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.306790 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.306759 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.308735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.308713 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.330893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.330869 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7gv\" (UniqueName: \"kubernetes.io/projected/c851ac2e-f840-4b78-bd26-85a3653ff657-kube-api-access-5w7gv\") pod \"maas-api-revoke-sub-4cde6729-klhdv\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.420796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.420753 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:33.724738 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:19:33.724707 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc851ac2e_f840_4b78_bd26_85a3653ff657.slice/crio-261e7214d0feb0de0d0895a87fcdb61df58677a391887348695ce8e8224eed18 WatchSource:0}: Error finding container 261e7214d0feb0de0d0895a87fcdb61df58677a391887348695ce8e8224eed18: Status 404 returned error can't find the container with id 261e7214d0feb0de0d0895a87fcdb61df58677a391887348695ce8e8224eed18 Nov 02 14:19:33.725273 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.725203 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv"] Nov 02 14:19:33.811952 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:33.811928 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" event={"ID":"c851ac2e-f840-4b78-bd26-85a3653ff657","Type":"ContainerStarted","Data":"261e7214d0feb0de0d0895a87fcdb61df58677a391887348695ce8e8224eed18"} Nov 02 14:19:34.821562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:34.821521 4431 generic.go:356] "Generic (PLEG): container finished" podID="c851ac2e-f840-4b78-bd26-85a3653ff657" containerID="b86c04d23a861a5959952c200d5e9e6b5d5a4841297b17a315c1777c7dc8f247" exitCode=0 Nov 02 14:19:34.822090 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:34.821599 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" event={"ID":"c851ac2e-f840-4b78-bd26-85a3653ff657","Type":"ContainerDied","Data":"b86c04d23a861a5959952c200d5e9e6b5d5a4841297b17a315c1777c7dc8f247"} Nov 02 14:19:35.035125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:35.035080 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757"] Nov 02 14:19:35.039376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:35.039352 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4171a261-kx757"] Nov 02 14:19:35.078025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:35.077998 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="626d7ce8-52ee-4ed3-9166-c2c4b4f82927" path="/var/lib/kubelet/pods/626d7ce8-52ee-4ed3-9166-c2c4b4f82927/volumes" Nov 02 14:19:35.977692 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:35.977665 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:36.040102 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.040071 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-openshift-ca\") pod \"c851ac2e-f840-4b78-bd26-85a3653ff657\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " Nov 02 14:19:36.040231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.040175 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-xks-ca\") pod \"c851ac2e-f840-4b78-bd26-85a3653ff657\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " Nov 02 14:19:36.040436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.040414 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c851ac2e-f840-4b78-bd26-85a3653ff657-kube-api-access-5w7gv\" (UniqueName: \"kubernetes.io/projected/c851ac2e-f840-4b78-bd26-85a3653ff657-kube-api-access-5w7gv\") pod \"c851ac2e-f840-4b78-bd26-85a3653ff657\" (UID: \"c851ac2e-f840-4b78-bd26-85a3653ff657\") " Nov 02 14:19:36.040505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.040474 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-openshift-ca" pod "c851ac2e-f840-4b78-bd26-85a3653ff657" (UID: "c851ac2e-f840-4b78-bd26-85a3653ff657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:36.040989 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.040964 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:36.042274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.042248 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-xks-ca" pod "c851ac2e-f840-4b78-bd26-85a3653ff657" (UID: "c851ac2e-f840-4b78-bd26-85a3653ff657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:19:36.042436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.042416 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c851ac2e-f840-4b78-bd26-85a3653ff657-kube-api-access-5w7gv" pod "c851ac2e-f840-4b78-bd26-85a3653ff657" (UID: "c851ac2e-f840-4b78-bd26-85a3653ff657"). InnerVolumeSpecName "kube-api-access-5w7gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:36.142399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.142373 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c851ac2e-f840-4b78-bd26-85a3653ff657-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:36.142565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.142402 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7gv\" (UniqueName: \"kubernetes.io/projected/c851ac2e-f840-4b78-bd26-85a3653ff657-kube-api-access-5w7gv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:36.835792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.835755 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" Nov 02 14:19:36.835961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.835786 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv" event={"ID":"c851ac2e-f840-4b78-bd26-85a3653ff657","Type":"ContainerDied","Data":"261e7214d0feb0de0d0895a87fcdb61df58677a391887348695ce8e8224eed18"} Nov 02 14:19:36.835961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:36.835847 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="261e7214d0feb0de0d0895a87fcdb61df58677a391887348695ce8e8224eed18" Nov 02 14:19:39.657865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:39.657834 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:19:45.077797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.077755 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd"] Nov 02 14:19:45.082699 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.082672 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d196c714-fw6cd"] Nov 02 14:19:45.404520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.404488 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974"] Nov 02 14:19:45.405389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.405358 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c851ac2e-f840-4b78-bd26-85a3653ff657" containerName="revoke-keys" Nov 02 14:19:45.405389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.405382 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c851ac2e-f840-4b78-bd26-85a3653ff657" containerName="revoke-keys" Nov 02 14:19:45.405650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.405632 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c851ac2e-f840-4b78-bd26-85a3653ff657" containerName="revoke-keys" Nov 02 14:19:45.409713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.409691 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.444345 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.444317 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974"] Nov 02 14:19:45.450129 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.450104 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:19:45.450300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.450206 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:19:45.563836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.563772 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.564017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.563867 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.564017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.563991 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqps6\" (UniqueName: \"kubernetes.io/projected/79d4cc7b-5c3e-4334-a353-295faba67545-kube-api-access-hqps6\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.664880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.664798 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.664880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.664846 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.665104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.664934 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqps6\" (UniqueName: \"kubernetes.io/projected/79d4cc7b-5c3e-4334-a353-295faba67545-kube-api-access-hqps6\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.665517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.665493 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.667659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.667637 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.689892 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.689868 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqps6\" (UniqueName: \"kubernetes.io/projected/79d4cc7b-5c3e-4334-a353-295faba67545-kube-api-access-hqps6\") pod \"maas-api-revoke-sub-711bc0df-z9974\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:45.724950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:45.724928 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:46.036812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:46.036770 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974"] Nov 02 14:19:46.038518 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:19:46.038476 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d4cc7b_5c3e_4334_a353_295faba67545.slice/crio-6f7e9fe6583e46a3716f2c419b599bf48723ecb1e98a3d60abf52e45b7b40068 WatchSource:0}: Error finding container 6f7e9fe6583e46a3716f2c419b599bf48723ecb1e98a3d60abf52e45b7b40068: Status 404 returned error can't find the container with id 6f7e9fe6583e46a3716f2c419b599bf48723ecb1e98a3d60abf52e45b7b40068 Nov 02 14:19:46.956595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:46.956562 4431 generic.go:356] "Generic (PLEG): container finished" podID="79d4cc7b-5c3e-4334-a353-295faba67545" containerID="c45b6fb8e4797d37792d175ae7fbe4fa37095c3aa903098f04a2b6075656903b" exitCode=0 Nov 02 14:19:46.956944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:46.956646 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" event={"ID":"79d4cc7b-5c3e-4334-a353-295faba67545","Type":"ContainerDied","Data":"c45b6fb8e4797d37792d175ae7fbe4fa37095c3aa903098f04a2b6075656903b"} Nov 02 14:19:46.956944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:46.956697 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" event={"ID":"79d4cc7b-5c3e-4334-a353-295faba67545","Type":"ContainerStarted","Data":"6f7e9fe6583e46a3716f2c419b599bf48723ecb1e98a3d60abf52e45b7b40068"} Nov 02 14:19:47.080056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:47.080020 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09243e92-8800-4b16-a99d-30da706f7fd4" path="/var/lib/kubelet/pods/09243e92-8800-4b16-a99d-30da706f7fd4/volumes" Nov 02 14:19:48.140235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.140204 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:48.299166 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.299091 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-xks-ca\") pod \"79d4cc7b-5c3e-4334-a353-295faba67545\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " Nov 02 14:19:48.299327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.299256 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d4cc7b-5c3e-4334-a353-295faba67545-kube-api-access-hqps6\" (UniqueName: \"kubernetes.io/projected/79d4cc7b-5c3e-4334-a353-295faba67545-kube-api-access-hqps6\") pod \"79d4cc7b-5c3e-4334-a353-295faba67545\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " Nov 02 14:19:48.299327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.299302 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-openshift-ca\") pod \"79d4cc7b-5c3e-4334-a353-295faba67545\" (UID: \"79d4cc7b-5c3e-4334-a353-295faba67545\") " Nov 02 14:19:48.299671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.299641 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-openshift-ca" pod "79d4cc7b-5c3e-4334-a353-295faba67545" (UID: "79d4cc7b-5c3e-4334-a353-295faba67545"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:19:48.301190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.301166 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-xks-ca" pod "79d4cc7b-5c3e-4334-a353-295faba67545" (UID: "79d4cc7b-5c3e-4334-a353-295faba67545"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:19:48.301448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.301360 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d4cc7b-5c3e-4334-a353-295faba67545-kube-api-access-hqps6" pod "79d4cc7b-5c3e-4334-a353-295faba67545" (UID: "79d4cc7b-5c3e-4334-a353-295faba67545"). InnerVolumeSpecName "kube-api-access-hqps6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:19:48.401104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.401078 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqps6\" (UniqueName: \"kubernetes.io/projected/79d4cc7b-5c3e-4334-a353-295faba67545-kube-api-access-hqps6\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:48.401104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.401103 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:48.401249 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.401112 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d4cc7b-5c3e-4334-a353-295faba67545-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:19:48.976851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.976821 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" Nov 02 14:19:48.976851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.976840 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974" event={"ID":"79d4cc7b-5c3e-4334-a353-295faba67545","Type":"ContainerDied","Data":"6f7e9fe6583e46a3716f2c419b599bf48723ecb1e98a3d60abf52e45b7b40068"} Nov 02 14:19:48.977058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:48.976909 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f7e9fe6583e46a3716f2c419b599bf48723ecb1e98a3d60abf52e45b7b40068" Nov 02 14:19:49.034456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:49.034433 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf"] Nov 02 14:19:49.040543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:49.040524 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9641e5ba-rlhgf"] Nov 02 14:19:49.076243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:49.076222 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6029d56a-5483-4a75-be08-bb6be38b21ac" path="/var/lib/kubelet/pods/6029d56a-5483-4a75-be08-bb6be38b21ac/volumes" Nov 02 14:19:51.299363 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:19:51.299316 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:19:57.398801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.397830 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs"] Nov 02 14:19:57.398801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.398736 4431 cpu_manager.go:414] "containerMap: removing container" podUID="79d4cc7b-5c3e-4334-a353-295faba67545" containerName="revoke-keys" Nov 02 14:19:57.398801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.398758 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d4cc7b-5c3e-4334-a353-295faba67545" containerName="revoke-keys" Nov 02 14:19:57.399413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.398965 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d4cc7b-5c3e-4334-a353-295faba67545" containerName="revoke-keys" Nov 02 14:19:57.414802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.412967 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs"] Nov 02 14:19:57.414802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.413891 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.446590 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.446559 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:19:57.446795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.446705 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:19:57.515036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.514999 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.515199 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.515050 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc7n5\" (UniqueName: \"kubernetes.io/projected/e8165b81-389a-4b7c-8d99-1f8a75f02988-kube-api-access-gc7n5\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.515253 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.515236 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.616101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.616065 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.616299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.616109 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc7n5\" (UniqueName: \"kubernetes.io/projected/e8165b81-389a-4b7c-8d99-1f8a75f02988-kube-api-access-gc7n5\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.616299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.616255 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.616715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.616688 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.618576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.618556 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.649638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.649580 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc7n5\" (UniqueName: \"kubernetes.io/projected/e8165b81-389a-4b7c-8d99-1f8a75f02988-kube-api-access-gc7n5\") pod \"maas-api-revoke-sub-a08d33ff-xwkbs\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:57.760631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:57.760609 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:19:58.065308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:58.065278 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs"] Nov 02 14:19:58.065669 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:19:58.065649 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8165b81_389a_4b7c_8d99_1f8a75f02988.slice/crio-2caca7628720d1f10a226483f8ebb7e1f3402f078cc76762aad365792ec63488 WatchSource:0}: Error finding container 2caca7628720d1f10a226483f8ebb7e1f3402f078cc76762aad365792ec63488: Status 404 returned error can't find the container with id 2caca7628720d1f10a226483f8ebb7e1f3402f078cc76762aad365792ec63488 Nov 02 14:19:59.033969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:59.033945 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj"] Nov 02 14:19:59.046554 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:59.046531 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4114ef0-fp7nj"] Nov 02 14:19:59.052802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:59.052755 4431 generic.go:356] "Generic (PLEG): container finished" podID="e8165b81-389a-4b7c-8d99-1f8a75f02988" containerID="83eae1dd9be85d4fa09507633084ed920ae20a2ffa68504f4241caa19628e520" exitCode=0 Nov 02 14:19:59.052886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:59.052807 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" event={"ID":"e8165b81-389a-4b7c-8d99-1f8a75f02988","Type":"ContainerDied","Data":"83eae1dd9be85d4fa09507633084ed920ae20a2ffa68504f4241caa19628e520"} Nov 02 14:19:59.052886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:59.052857 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" event={"ID":"e8165b81-389a-4b7c-8d99-1f8a75f02988","Type":"ContainerStarted","Data":"2caca7628720d1f10a226483f8ebb7e1f3402f078cc76762aad365792ec63488"} Nov 02 14:19:59.076889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:19:59.076865 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed539f16-e6dc-4389-a554-6a36eb55b0a8" path="/var/lib/kubelet/pods/ed539f16-e6dc-4389-a554-6a36eb55b0a8/volumes" Nov 02 14:20:00.221143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.221117 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:20:00.350119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.350029 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-openshift-ca\") pod \"e8165b81-389a-4b7c-8d99-1f8a75f02988\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " Nov 02 14:20:00.350313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.350128 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8165b81-389a-4b7c-8d99-1f8a75f02988-kube-api-access-gc7n5\" (UniqueName: \"kubernetes.io/projected/e8165b81-389a-4b7c-8d99-1f8a75f02988-kube-api-access-gc7n5\") pod \"e8165b81-389a-4b7c-8d99-1f8a75f02988\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " Nov 02 14:20:00.350313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.350189 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-xks-ca\") pod \"e8165b81-389a-4b7c-8d99-1f8a75f02988\" (UID: \"e8165b81-389a-4b7c-8d99-1f8a75f02988\") " Nov 02 14:20:00.350466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.350428 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-openshift-ca" pod "e8165b81-389a-4b7c-8d99-1f8a75f02988" (UID: "e8165b81-389a-4b7c-8d99-1f8a75f02988"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:00.350905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.350880 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:20:00.352284 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.352258 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-xks-ca" pod "e8165b81-389a-4b7c-8d99-1f8a75f02988" (UID: "e8165b81-389a-4b7c-8d99-1f8a75f02988"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:20:00.352481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.352458 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8165b81-389a-4b7c-8d99-1f8a75f02988-kube-api-access-gc7n5" pod "e8165b81-389a-4b7c-8d99-1f8a75f02988" (UID: "e8165b81-389a-4b7c-8d99-1f8a75f02988"). InnerVolumeSpecName "kube-api-access-gc7n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:00.452168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.452141 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc7n5\" (UniqueName: \"kubernetes.io/projected/e8165b81-389a-4b7c-8d99-1f8a75f02988-kube-api-access-gc7n5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:20:00.452168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:00.452168 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165b81-389a-4b7c-8d99-1f8a75f02988-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:20:01.069065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:01.069038 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" Nov 02 14:20:01.075521 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:01.075491 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs" event={"ID":"e8165b81-389a-4b7c-8d99-1f8a75f02988","Type":"ContainerDied","Data":"2caca7628720d1f10a226483f8ebb7e1f3402f078cc76762aad365792ec63488"} Nov 02 14:20:01.075657 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:01.075532 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2caca7628720d1f10a226483f8ebb7e1f3402f078cc76762aad365792ec63488" Nov 02 14:20:15.986386 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:15.986350 4431 scope.go:122] "RemoveContainer" containerID="44d1b1bb581fb9c4b00feade42ce1d470ff2b2f3f4ac2b4baee87b7f6a0e7954" Nov 02 14:20:15.996399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:15.996379 4431 scope.go:122] "RemoveContainer" containerID="8540db92fb944151846406e695f52737d983e492a8bef16cd266e4893fbd3878" Nov 02 14:20:21.285123 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:20:21.285071 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:20:24.033083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:24.033046 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv"] Nov 02 14:20:24.045881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:24.045857 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ad324bb-5zvhv"] Nov 02 14:20:25.077914 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:25.077884 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed6bc10-83ff-481a-abe5-f0732aef7e05" path="/var/lib/kubelet/pods/9ed6bc10-83ff-481a-abe5-f0732aef7e05/volumes" Nov 02 14:20:31.239935 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:31.239843 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:20:34.042043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:34.042013 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk"] Nov 02 14:20:34.054857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:34.054828 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29cc4291-759lk"] Nov 02 14:20:35.040194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:35.040161 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r"] Nov 02 14:20:35.053279 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:35.053255 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e5428c7-rcv2r"] Nov 02 14:20:35.077346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:35.077322 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5961c0c8-a2d3-48e2-95ce-7ae99ec24e23" path="/var/lib/kubelet/pods/5961c0c8-a2d3-48e2-95ce-7ae99ec24e23/volumes" Nov 02 14:20:35.077873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:35.077857 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a162de50-020e-4e2b-8c42-e0145358ca83" path="/var/lib/kubelet/pods/a162de50-020e-4e2b-8c42-e0145358ca83/volumes" Nov 02 14:20:44.658398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.658368 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd"] Nov 02 14:20:44.659031 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.659007 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e8165b81-389a-4b7c-8d99-1f8a75f02988" containerName="revoke-keys" Nov 02 14:20:44.659031 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.659030 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8165b81-389a-4b7c-8d99-1f8a75f02988" containerName="revoke-keys" Nov 02 14:20:44.659238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.659221 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8165b81-389a-4b7c-8d99-1f8a75f02988" containerName="revoke-keys" Nov 02 14:20:44.662755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.662733 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.679600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.679578 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c9fc7d82-openshift-default-dockercfg-4d9zb\"" Nov 02 14:20:44.697421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.697389 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlj7s\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-kube-api-access-xlj7s\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.697655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.697633 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-token\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.697808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.697466 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd"] Nov 02 14:20:44.697984 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.697962 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-certs\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.698114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.698002 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-envoy\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.698192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.698177 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-credential-socket\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.698254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.698201 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-podinfo\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.698254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.698224 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-data\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.698254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.698248 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8ff025-8014-4768-b7d3-37c10d3471f8-istiod-ca-cert\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.698417 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.698372 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-socket\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.803698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.803663 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlj7s\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-kube-api-access-xlj7s\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.804017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.803993 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-token\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.804244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.804225 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-certs\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.804404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.804388 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-envoy\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.805334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.805289 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-credential-socket\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.805503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.805485 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-podinfo\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.805638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.805610 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-data\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.805757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.805742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8ff025-8014-4768-b7d3-37c10d3471f8-istiod-ca-cert\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.805885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.805870 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-socket\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.806493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.806470 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-socket\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.807395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.807363 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-certs\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.808450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.808426 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-credential-socket\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.819663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.819345 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8ff025-8014-4768-b7d3-37c10d3471f8-istiod-ca-cert\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.819859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.819605 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-data\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.837404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.837347 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-podinfo\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.862940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.862880 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlj7s\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-kube-api-access-xlj7s\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.868838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.863391 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-envoy\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.873188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.873160 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-token\") pod \"e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:44.981709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:44.981619 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:45.328011 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:20:45.327981 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8ff025_8014_4768_b7d3_37c10d3471f8.slice/crio-c631b008b88107e68e9517d9aee8f773861e6741ef4ad9fda0c1e8a5af963f33 WatchSource:0}: Error finding container c631b008b88107e68e9517d9aee8f773861e6741ef4ad9fda0c1e8a5af963f33: Status 404 returned error can't find the container with id c631b008b88107e68e9517d9aee8f773861e6741ef4ad9fda0c1e8a5af963f33 Nov 02 14:20:45.328594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:45.328568 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd"] Nov 02 14:20:45.330493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:45.330457 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:20:45.330584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:45.330523 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:20:45.330584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:45.330553 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:20:45.412760 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:45.412732 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" event={"ID":"2d8ff025-8014-4768-b7d3-37c10d3471f8","Type":"ContainerStarted","Data":"c631b008b88107e68e9517d9aee8f773861e6741ef4ad9fda0c1e8a5af963f33"} Nov 02 14:20:46.421024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:46.420982 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" event={"ID":"2d8ff025-8014-4768-b7d3-37c10d3471f8","Type":"ContainerStarted","Data":"90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb"} Nov 02 14:20:46.499320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:46.499269 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" podStartSLOduration=2.499255731 podStartE2EDuration="2.499255731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:46.484431276 +0000 UTC m=+1411.521536207" watchObservedRunningTime="2026-11-02 14:20:46.499255731 +0000 UTC m=+1411.536360662" Nov 02 14:20:46.982289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:46.982260 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:46.987637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:46.987613 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:47.429601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:47.429572 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:47.430589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:47.430571 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:20:49.034015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:49.033979 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm"] Nov 02 14:20:49.047154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:49.047124 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75"] Nov 02 14:20:49.051049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:49.051023 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8475d54-2w6dm"] Nov 02 14:20:49.054344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:49.054323 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72df75c8-x6v75"] Nov 02 14:20:49.075663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:49.075635 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef" path="/var/lib/kubelet/pods/0d66b80d-f801-49cb-aa1c-9ab9e4fe85ef/volumes" Nov 02 14:20:49.076136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:49.076122 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf" path="/var/lib/kubelet/pods/1bc76f74-f1b7-4c92-af67-3e21a2b2a4bf/volumes" Nov 02 14:20:51.284961 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:20:51.284926 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:20:53.506478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.506448 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl"] Nov 02 14:20:53.512983 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.512951 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.532438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.532412 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl"] Nov 02 14:20:53.537186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.537163 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:20:53.537353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.537229 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:20:53.612970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.612946 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.613104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.612994 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.613104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.613082 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7j5\" (UniqueName: \"kubernetes.io/projected/ef38aff1-2a98-440c-869a-f6f5570f868b-kube-api-access-pm7j5\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.714873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.714841 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7j5\" (UniqueName: \"kubernetes.io/projected/ef38aff1-2a98-440c-869a-f6f5570f868b-kube-api-access-pm7j5\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.715053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.715036 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.715110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.715094 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.715735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.715712 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.717435 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.717418 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.739278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.739257 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7j5\" (UniqueName: \"kubernetes.io/projected/ef38aff1-2a98-440c-869a-f6f5570f868b-kube-api-access-pm7j5\") pod \"maas-api-revoke-sub-9867ab2a-gfnfl\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:53.829950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:53.829899 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:54.137320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:54.137275 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl"] Nov 02 14:20:54.137446 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:20:54.137379 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef38aff1_2a98_440c_869a_f6f5570f868b.slice/crio-5035087e7c58ba1352633c1a88185cd181881e1047d94240863aa625c27f2e7a WatchSource:0}: Error finding container 5035087e7c58ba1352633c1a88185cd181881e1047d94240863aa625c27f2e7a: Status 404 returned error can't find the container with id 5035087e7c58ba1352633c1a88185cd181881e1047d94240863aa625c27f2e7a Nov 02 14:20:54.483275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:54.483248 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" event={"ID":"ef38aff1-2a98-440c-869a-f6f5570f868b","Type":"ContainerStarted","Data":"5035087e7c58ba1352633c1a88185cd181881e1047d94240863aa625c27f2e7a"} Nov 02 14:20:54.675827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:54.675703 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:20:55.492419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:55.492380 4431 generic.go:356] "Generic (PLEG): container finished" podID="ef38aff1-2a98-440c-869a-f6f5570f868b" containerID="9cf551715f6c97b36208ac190e57606d1e5592187c6d2b44c738ef405e5c7af3" exitCode=0 Nov 02 14:20:55.492946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:55.492468 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" event={"ID":"ef38aff1-2a98-440c-869a-f6f5570f868b","Type":"ContainerDied","Data":"9cf551715f6c97b36208ac190e57606d1e5592187c6d2b44c738ef405e5c7af3"} Nov 02 14:20:56.687318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.686917 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:56.771880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.771800 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-openshift-ca\") pod \"ef38aff1-2a98-440c-869a-f6f5570f868b\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " Nov 02 14:20:56.772069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.772001 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef38aff1-2a98-440c-869a-f6f5570f868b-kube-api-access-pm7j5\" (UniqueName: \"kubernetes.io/projected/ef38aff1-2a98-440c-869a-f6f5570f868b-kube-api-access-pm7j5\") pod \"ef38aff1-2a98-440c-869a-f6f5570f868b\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " Nov 02 14:20:56.772069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.772033 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-xks-ca\") pod \"ef38aff1-2a98-440c-869a-f6f5570f868b\" (UID: \"ef38aff1-2a98-440c-869a-f6f5570f868b\") " Nov 02 14:20:56.772266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.772237 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-openshift-ca" pod "ef38aff1-2a98-440c-869a-f6f5570f868b" (UID: "ef38aff1-2a98-440c-869a-f6f5570f868b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:20:56.772503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.772482 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:20:56.774269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.774243 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-xks-ca" pod "ef38aff1-2a98-440c-869a-f6f5570f868b" (UID: "ef38aff1-2a98-440c-869a-f6f5570f868b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:20:56.774374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.774269 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef38aff1-2a98-440c-869a-f6f5570f868b-kube-api-access-pm7j5" pod "ef38aff1-2a98-440c-869a-f6f5570f868b" (UID: "ef38aff1-2a98-440c-869a-f6f5570f868b"). InnerVolumeSpecName "kube-api-access-pm7j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:20:56.873598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.873562 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm7j5\" (UniqueName: \"kubernetes.io/projected/ef38aff1-2a98-440c-869a-f6f5570f868b-kube-api-access-pm7j5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:20:56.873598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:56.873592 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef38aff1-2a98-440c-869a-f6f5570f868b-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:20:57.510065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:57.510039 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" Nov 02 14:20:57.510248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:57.510074 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl" event={"ID":"ef38aff1-2a98-440c-869a-f6f5570f868b","Type":"ContainerDied","Data":"5035087e7c58ba1352633c1a88185cd181881e1047d94240863aa625c27f2e7a"} Nov 02 14:20:57.510248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:20:57.510128 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5035087e7c58ba1352633c1a88185cd181881e1047d94240863aa625c27f2e7a" Nov 02 14:21:00.042293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:00.042262 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7"] Nov 02 14:21:00.055181 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:00.055151 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f56c5b-7mjm7"] Nov 02 14:21:01.076383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:01.076351 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c34465-9e33-4a84-9182-b93afb13b32a" path="/var/lib/kubelet/pods/04c34465-9e33-4a84-9182-b93afb13b32a/volumes" Nov 02 14:21:04.031650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:04.031612 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v"] Nov 02 14:21:04.044459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:04.044433 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2ad82-k2w8v"] Nov 02 14:21:05.077558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:05.077524 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0d6682-a02c-4351-9d73-35f0dd936b60" path="/var/lib/kubelet/pods/fd0d6682-a02c-4351-9d73-35f0dd936b60/volumes" Nov 02 14:21:13.006550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:13.006518 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq"] Nov 02 14:21:13.845275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:13.844714 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294"] Nov 02 14:21:16.113811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.110889 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:21:16.113811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.110962 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:16.133565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.122971 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:21:16.133565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.123027 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:16.205857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.205827 4431 scope.go:122] "RemoveContainer" containerID="146de96eb84824d2cfd463f2102abeeacfa8e72d5969b06112dbf9285536f7a1" Nov 02 14:21:16.367964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.367927 4431 scope.go:122] "RemoveContainer" containerID="4f98bc4ec99c30a9362442030d7f3abd45a512d6a4c6a8f8f772570dc09ddd46" Nov 02 14:21:16.385933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.385616 4431 scope.go:122] "RemoveContainer" containerID="912bc5486debd14728fef4c588c675cc44170629b9889b4a5432c1ac2b6e0101" Nov 02 14:21:16.404270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.403825 4431 scope.go:122] "RemoveContainer" containerID="c062f3ead6e82095cba0260fbba3bf58f4427b48315b8a0c4e139df583cdf455" Nov 02 14:21:16.418916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:16.418594 4431 scope.go:122] "RemoveContainer" containerID="2f7461d3b8aea716628bd434bfb640d5e14dc27d0767bfc10a251cceeb434c4b" Nov 02 14:21:18.007461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.007413 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" podUID="e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" containerName="payload-processing" containerID="cri-o://933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6" gracePeriod=25 Nov 02 14:21:18.273341 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.273286 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:21:18.401939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.401911 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-plugins-config-volume\") pod \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " Nov 02 14:21:18.402077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.401978 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-provider-credentials\") pod \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " Nov 02 14:21:18.402178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.402161 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-kube-api-access-4lf4t\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-kube-api-access-4lf4t\") pod \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " Nov 02 14:21:18.402305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.402289 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-routing-overlay-volume\") pod \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\" (UID: \"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa\") " Nov 02 14:21:18.402376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.402357 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-plugins-config-volume" pod "e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" (UID: "e068bff5-9b0b-48c3-8d6b-54ee3c117ffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:18.402668 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.402639 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-routing-overlay-volume" pod "e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" (UID: "e068bff5-9b0b-48c3-8d6b-54ee3c117ffa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:18.402795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.402710 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:18.402795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.402726 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:18.404164 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.404142 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-provider-credentials" pod "e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" (UID: "e068bff5-9b0b-48c3-8d6b-54ee3c117ffa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:18.404261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.404209 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-kube-api-access-4lf4t" pod "e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" (UID: "e068bff5-9b0b-48c3-8d6b-54ee3c117ffa"). InnerVolumeSpecName "kube-api-access-4lf4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:18.503719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.503690 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lf4t\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-kube-api-access-4lf4t\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:18.503719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.503717 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:18.742138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.742098 4431 generic.go:356] "Generic (PLEG): container finished" podID="e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" containerID="933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6" exitCode=0 Nov 02 14:21:18.742316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.742136 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" event={"ID":"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa","Type":"ContainerDied","Data":"933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6"} Nov 02 14:21:18.742316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.742146 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" Nov 02 14:21:18.742316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.742192 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq" event={"ID":"e068bff5-9b0b-48c3-8d6b-54ee3c117ffa","Type":"ContainerDied","Data":"f5c10690d82634423a1dc0412c847039699ed43ca5d5accd732ac4eeb5b4eac7"} Nov 02 14:21:18.742316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.742215 4431 scope.go:122] "RemoveContainer" containerID="933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6" Nov 02 14:21:18.753179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.753162 4431 scope.go:122] "RemoveContainer" containerID="933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6" Nov 02 14:21:18.753435 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:21:18.753420 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6\": container with ID starting with 933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6 not found: ID does not exist" containerID="933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6" Nov 02 14:21:18.753475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.753443 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6"} err="failed to get container status \"933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6\": rpc error: code = NotFound desc = could not find container \"933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6\": container with ID starting with 933ce96e58850dae91343bc1d6afc22a3dbd521902ce80d6f174895660564bb6 not found: ID does not exist" Nov 02 14:21:18.800674 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.800647 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq"] Nov 02 14:21:18.812992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.812967 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6r67kq"] Nov 02 14:21:18.845913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:18.845882 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" podUID="d0334c06-78a9-4b73-9f96-3cfa5256a860" containerName="payload-processing" containerID="cri-o://5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25" gracePeriod=25 Nov 02 14:21:19.041643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.041621 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8"] Nov 02 14:21:19.047631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.047604 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11da6b8c-7npc8"] Nov 02 14:21:19.078565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.078534 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fdcd30-9ebd-404c-a99f-bd9bba80b888" path="/var/lib/kubelet/pods/22fdcd30-9ebd-404c-a99f-bd9bba80b888/volumes" Nov 02 14:21:19.079027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.079009 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" path="/var/lib/kubelet/pods/e068bff5-9b0b-48c3-8d6b-54ee3c117ffa/volumes" Nov 02 14:21:19.132519 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.132498 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:21:19.211077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.211049 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-routing-overlay-volume\") pod \"d0334c06-78a9-4b73-9f96-3cfa5256a860\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " Nov 02 14:21:19.211267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.211250 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-provider-credentials\") pod \"d0334c06-78a9-4b73-9f96-3cfa5256a860\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " Nov 02 14:21:19.211334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.211284 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-plugins-config-volume\") pod \"d0334c06-78a9-4b73-9f96-3cfa5256a860\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " Nov 02 14:21:19.211334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.211314 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-kube-api-access-477lm\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-kube-api-access-477lm\") pod \"d0334c06-78a9-4b73-9f96-3cfa5256a860\" (UID: \"d0334c06-78a9-4b73-9f96-3cfa5256a860\") " Nov 02 14:21:19.211513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.211483 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-routing-overlay-volume" pod "d0334c06-78a9-4b73-9f96-3cfa5256a860" (UID: "d0334c06-78a9-4b73-9f96-3cfa5256a860"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:19.211746 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.211724 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-plugins-config-volume" pod "d0334c06-78a9-4b73-9f96-3cfa5256a860" (UID: "d0334c06-78a9-4b73-9f96-3cfa5256a860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:19.212082 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.212058 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:19.212186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.212087 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d0334c06-78a9-4b73-9f96-3cfa5256a860-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:19.213587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.213558 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-kube-api-access-477lm" pod "d0334c06-78a9-4b73-9f96-3cfa5256a860" (UID: "d0334c06-78a9-4b73-9f96-3cfa5256a860"). InnerVolumeSpecName "kube-api-access-477lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:19.213681 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.213587 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-provider-credentials" pod "d0334c06-78a9-4b73-9f96-3cfa5256a860" (UID: "d0334c06-78a9-4b73-9f96-3cfa5256a860"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:19.313470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.313396 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:19.313470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.313421 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-477lm\" (UniqueName: \"kubernetes.io/projected/d0334c06-78a9-4b73-9f96-3cfa5256a860-kube-api-access-477lm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:19.750636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.750598 4431 generic.go:356] "Generic (PLEG): container finished" podID="d0334c06-78a9-4b73-9f96-3cfa5256a860" containerID="5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25" exitCode=0 Nov 02 14:21:19.750832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.750649 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" Nov 02 14:21:19.750832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.750689 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" event={"ID":"d0334c06-78a9-4b73-9f96-3cfa5256a860","Type":"ContainerDied","Data":"5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25"} Nov 02 14:21:19.750832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.750748 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294" event={"ID":"d0334c06-78a9-4b73-9f96-3cfa5256a860","Type":"ContainerDied","Data":"9c3318d38f60288b20b6f80301db8b88c049299d7e87d446203a9bb0243b87a8"} Nov 02 14:21:19.750832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.750769 4431 scope.go:122] "RemoveContainer" containerID="5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25" Nov 02 14:21:19.761507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.761490 4431 scope.go:122] "RemoveContainer" containerID="5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25" Nov 02 14:21:19.761769 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:21:19.761751 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25\": container with ID starting with 5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25 not found: ID does not exist" containerID="5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25" Nov 02 14:21:19.761844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.761810 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25"} err="failed to get container status \"5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25\": rpc error: code = NotFound desc = could not find container \"5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25\": container with ID starting with 5b73b3dbdcea8a4f066ec94312040784f0c741e19dd0aee6b76ff450e6cadc25 not found: ID does not exist" Nov 02 14:21:19.809994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.809973 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294"] Nov 02 14:21:19.823013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:19.822989 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fdc294"] Nov 02 14:21:21.078203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:21.078163 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0334c06-78a9-4b73-9f96-3cfa5256a860" path="/var/lib/kubelet/pods/d0334c06-78a9-4b73-9f96-3cfa5256a860/volumes" Nov 02 14:21:21.286722 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:21:21.286686 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:21:24.032211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:24.032178 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r"] Nov 02 14:21:24.045318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:24.045292 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff2806df-2vr2r"] Nov 02 14:21:25.077943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:25.077907 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5240b296-8225-47de-b5b9-18a282d82a95" path="/var/lib/kubelet/pods/5240b296-8225-47de-b5b9-18a282d82a95/volumes" Nov 02 14:21:25.840634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:25.840593 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:21:34.815427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.815393 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4"] Nov 02 14:21:34.815944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.815931 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d0334c06-78a9-4b73-9f96-3cfa5256a860" containerName="payload-processing" Nov 02 14:21:34.815995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.815946 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0334c06-78a9-4b73-9f96-3cfa5256a860" containerName="payload-processing" Nov 02 14:21:34.815995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.815977 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" containerName="payload-processing" Nov 02 14:21:34.815995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.815984 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" containerName="payload-processing" Nov 02 14:21:34.816124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.815999 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ef38aff1-2a98-440c-869a-f6f5570f868b" containerName="revoke-keys" Nov 02 14:21:34.816124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.816004 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef38aff1-2a98-440c-869a-f6f5570f868b" containerName="revoke-keys" Nov 02 14:21:34.816124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.816104 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0334c06-78a9-4b73-9f96-3cfa5256a860" containerName="payload-processing" Nov 02 14:21:34.816124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.816117 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e068bff5-9b0b-48c3-8d6b-54ee3c117ffa" containerName="payload-processing" Nov 02 14:21:34.816124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.816124 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef38aff1-2a98-440c-869a-f6f5570f868b" containerName="revoke-keys" Nov 02 14:21:34.821114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.821096 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:34.842269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.842243 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4"] Nov 02 14:21:34.851025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.851003 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:21:34.851160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.851093 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:21:34.955871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.955833 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pktd\" (UniqueName: \"kubernetes.io/projected/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-kube-api-access-7pktd\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:34.956027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.955908 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:34.956027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:34.956014 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:35.056648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.056619 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pktd\" (UniqueName: \"kubernetes.io/projected/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-kube-api-access-7pktd\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:35.056851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.056663 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:35.056851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.056702 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:35.057391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.057365 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:35.059141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.059118 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:35.085686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.085571 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pktd\" (UniqueName: \"kubernetes.io/projected/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-kube-api-access-7pktd\") pod \"maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:35.139318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.139293 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:35.445062 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:21:35.445031 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ea4e45_1b6c_4ea1_9b25_7656a9f9f9fe.slice/crio-84f2b545fcce3856446239cfb1d38dee123cd765ec5997b85f30f4c1e126388e WatchSource:0}: Error finding container 84f2b545fcce3856446239cfb1d38dee123cd765ec5997b85f30f4c1e126388e: Status 404 returned error can't find the container with id 84f2b545fcce3856446239cfb1d38dee123cd765ec5997b85f30f4c1e126388e Nov 02 14:21:35.446562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.446531 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4"] Nov 02 14:21:35.940787 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.940747 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" event={"ID":"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe","Type":"ContainerStarted","Data":"b6854044271b1d79379a74cb8eda5ab4375d80fc4c09082b6d71934e8b31c4d0"} Nov 02 14:21:35.943591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.940824 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" event={"ID":"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe","Type":"ContainerStarted","Data":"84f2b545fcce3856446239cfb1d38dee123cd765ec5997b85f30f4c1e126388e"} Nov 02 14:21:35.987226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:35.987170 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" podStartSLOduration=1.9871521030000001 podStartE2EDuration="1.987152103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:35.984000199 +0000 UTC m=+1461.021105130" watchObservedRunningTime="2026-11-02 14:21:35.987152103 +0000 UTC m=+1461.024257035" Nov 02 14:21:36.053112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.053077 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f"] Nov 02 14:21:36.056672 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.056650 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7135e531-4r78f"] Nov 02 14:21:36.949217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.949177 4431 generic.go:356] "Generic (PLEG): container finished" podID="f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe" containerID="b6854044271b1d79379a74cb8eda5ab4375d80fc4c09082b6d71934e8b31c4d0" exitCode=0 Nov 02 14:21:36.949619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.949255 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" event={"ID":"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe","Type":"ContainerDied","Data":"b6854044271b1d79379a74cb8eda5ab4375d80fc4c09082b6d71934e8b31c4d0"} Nov 02 14:21:36.950959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.950941 4431 generic.go:356] "Generic (PLEG): container finished" podID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerID="90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb" exitCode=0 Nov 02 14:21:36.951032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.951009 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" event={"ID":"2d8ff025-8014-4768-b7d3-37c10d3471f8","Type":"ContainerDied","Data":"90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb"} Nov 02 14:21:36.951361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.951347 4431 scope.go:122] "RemoveContainer" containerID="90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb" Nov 02 14:21:36.952765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.952732 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:21:36.952861 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.952831 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:21:36.952902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:36.952869 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:21:37.081939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:37.081908 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19042772-9e2a-4685-b598-6bef25619c28" path="/var/lib/kubelet/pods/19042772-9e2a-4685-b598-6bef25619c28/volumes" Nov 02 14:21:37.959707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:37.959671 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" event={"ID":"2d8ff025-8014-4768-b7d3-37c10d3471f8","Type":"ContainerStarted","Data":"56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49"} Nov 02 14:21:37.996802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:37.991642 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:21:38.212992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.212929 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:38.309385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.309290 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-kube-api-access-7pktd\" (UniqueName: \"kubernetes.io/projected/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-kube-api-access-7pktd\") pod \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " Nov 02 14:21:38.310227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.310199 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-xks-ca\") pod \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " Nov 02 14:21:38.310456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.310438 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-openshift-ca\") pod \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\" (UID: \"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe\") " Nov 02 14:21:38.310916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.310879 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-openshift-ca" pod "f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe" (UID: "f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:38.311303 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.311277 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:38.313093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.313070 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-kube-api-access-7pktd" pod "f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe" (UID: "f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe"). InnerVolumeSpecName "kube-api-access-7pktd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:38.313761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.313732 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-xks-ca" pod "f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe" (UID: "f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:21:38.414804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.412970 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pktd\" (UniqueName: \"kubernetes.io/projected/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-kube-api-access-7pktd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:38.414804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.413011 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:38.981801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.980797 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" Nov 02 14:21:38.996797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.984332 4431 patch_prober.go:29] interesting pod/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 14:21:38.996797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.984399 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 14:21:38.996797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.984817 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4" event={"ID":"f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe","Type":"ContainerDied","Data":"84f2b545fcce3856446239cfb1d38dee123cd765ec5997b85f30f4c1e126388e"} Nov 02 14:21:38.996797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:38.984881 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84f2b545fcce3856446239cfb1d38dee123cd765ec5997b85f30f4c1e126388e" Nov 02 14:21:39.984801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:39.984133 4431 patch_prober.go:29] interesting pod/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 14:21:39.984801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:39.984196 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 14:21:40.687801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:40.687622 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:40.714893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:40.714848 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:40.796802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:40.787040 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:40.987370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:40.987281 4431 patch_prober.go:29] interesting pod/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 14:21:40.987983 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:40.987944 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 14:21:40.995909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:40.995864 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:41.092807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.092469 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/health\": context deadline exceeded" Nov 02 14:21:41.183801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.176849 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:41.226801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.222839 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 02 14:21:41.291856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.290425 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": context deadline exceeded" Nov 02 14:21:41.299205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.298849 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:41.494944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.493861 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Nov 02 14:21:41.525800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.517988 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:41.689801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.687855 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Nov 02 14:21:41.689801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.688665 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:21:41.800811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.791871 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:41.800811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:41.792333 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:21:42.010637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.010536 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:42.053479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.029438 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:21:42.163680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.163329 4431 patch_prober.go:29] interesting pod/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 14:21:42.163989 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.163948 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 14:21:42.245796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.235871 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:21:42.245796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.236506 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:21:42.274874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.250824 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:21:42.313182 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.313023 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:21:42.352868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.350491 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:21:42.365837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.364462 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:21:42.995995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.995949 4431 patch_prober.go:29] interesting pod/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 02 14:21:42.996185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:42.996024 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 02 14:21:43.986704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:43.986674 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:21:43.987214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:43.987151 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:21:43.987768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:43.987752 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:21:44.709610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.709578 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj"] Nov 02 14:21:44.710149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.710135 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe" containerName="revoke-keys" Nov 02 14:21:44.710199 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.710151 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe" containerName="revoke-keys" Nov 02 14:21:44.710278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.710269 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe" containerName="revoke-keys" Nov 02 14:21:44.712744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.712721 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.733098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.733077 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj"] Nov 02 14:21:44.741894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.741862 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:21:44.742029 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.741935 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:21:44.855856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.855830 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.856019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.855885 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwfgb\" (UniqueName: \"kubernetes.io/projected/540f1c37-7470-43e5-9f55-6b84b0d5ad18-kube-api-access-dwfgb\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.856069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.856018 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.957216 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.957189 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.957366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.957240 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwfgb\" (UniqueName: \"kubernetes.io/projected/540f1c37-7470-43e5-9f55-6b84b0d5ad18-kube-api-access-dwfgb\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.957366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.957313 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.957986 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.957967 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.959542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.959525 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:44.982313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:44.982259 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwfgb\" (UniqueName: \"kubernetes.io/projected/540f1c37-7470-43e5-9f55-6b84b0d5ad18-kube-api-access-dwfgb\") pod \"maas-api-revoke-sub-27fcdadf-gdhlj\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:45.025667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:45.025649 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:45.049131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:45.049108 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm"] Nov 02 14:21:45.055277 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:45.055255 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-877c2007-j2fgm"] Nov 02 14:21:45.107192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:45.103088 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3092e93f-2bef-4d31-b63b-355e989d5b18" path="/var/lib/kubelet/pods/3092e93f-2bef-4d31-b63b-355e989d5b18/volumes" Nov 02 14:21:45.344813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:45.344751 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj"] Nov 02 14:21:45.344946 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:21:45.344749 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540f1c37_7470_43e5_9f55_6b84b0d5ad18.slice/crio-0815eee48859c7200fde6af67f9f1951a352cc86bd7cf6e9de42894982a2e77d WatchSource:0}: Error finding container 0815eee48859c7200fde6af67f9f1951a352cc86bd7cf6e9de42894982a2e77d: Status 404 returned error can't find the container with id 0815eee48859c7200fde6af67f9f1951a352cc86bd7cf6e9de42894982a2e77d Nov 02 14:21:46.338360 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:46.338322 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" event={"ID":"540f1c37-7470-43e5-9f55-6b84b0d5ad18","Type":"ContainerStarted","Data":"05d9487656395b36d88b1d5d009d9094e80eab5fc0181dc0645d9d08973fc193"} Nov 02 14:21:46.338896 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:46.338383 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" event={"ID":"540f1c37-7470-43e5-9f55-6b84b0d5ad18","Type":"ContainerStarted","Data":"0815eee48859c7200fde6af67f9f1951a352cc86bd7cf6e9de42894982a2e77d"} Nov 02 14:21:46.385950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:46.385897 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" podStartSLOduration=2.385882729 podStartE2EDuration="2.385882729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:46.38125501 +0000 UTC m=+1471.418359942" watchObservedRunningTime="2026-11-02 14:21:46.385882729 +0000 UTC m=+1471.422987661" Nov 02 14:21:47.347527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:47.347494 4431 generic.go:356] "Generic (PLEG): container finished" podID="540f1c37-7470-43e5-9f55-6b84b0d5ad18" containerID="05d9487656395b36d88b1d5d009d9094e80eab5fc0181dc0645d9d08973fc193" exitCode=0 Nov 02 14:21:47.347527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:47.347529 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" event={"ID":"540f1c37-7470-43e5-9f55-6b84b0d5ad18","Type":"ContainerDied","Data":"05d9487656395b36d88b1d5d009d9094e80eab5fc0181dc0645d9d08973fc193"} Nov 02 14:21:48.516147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.516124 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:48.620377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.620300 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-xks-ca\") pod \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " Nov 02 14:21:48.620545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.620487 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-openshift-ca\") pod \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " Nov 02 14:21:48.620545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.620537 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540f1c37-7470-43e5-9f55-6b84b0d5ad18-kube-api-access-dwfgb\" (UniqueName: \"kubernetes.io/projected/540f1c37-7470-43e5-9f55-6b84b0d5ad18-kube-api-access-dwfgb\") pod \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\" (UID: \"540f1c37-7470-43e5-9f55-6b84b0d5ad18\") " Nov 02 14:21:48.620842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.620820 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-openshift-ca" pod "540f1c37-7470-43e5-9f55-6b84b0d5ad18" (UID: "540f1c37-7470-43e5-9f55-6b84b0d5ad18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:21:48.621009 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.620984 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:48.622370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.622346 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-xks-ca" pod "540f1c37-7470-43e5-9f55-6b84b0d5ad18" (UID: "540f1c37-7470-43e5-9f55-6b84b0d5ad18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:21:48.622561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.622538 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540f1c37-7470-43e5-9f55-6b84b0d5ad18-kube-api-access-dwfgb" pod "540f1c37-7470-43e5-9f55-6b84b0d5ad18" (UID: "540f1c37-7470-43e5-9f55-6b84b0d5ad18"). InnerVolumeSpecName "kube-api-access-dwfgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:21:48.722437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.722411 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwfgb\" (UniqueName: \"kubernetes.io/projected/540f1c37-7470-43e5-9f55-6b84b0d5ad18-kube-api-access-dwfgb\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:48.722437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:48.722435 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540f1c37-7470-43e5-9f55-6b84b0d5ad18-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:21:49.363221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:49.363147 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" event={"ID":"540f1c37-7470-43e5-9f55-6b84b0d5ad18","Type":"ContainerDied","Data":"0815eee48859c7200fde6af67f9f1951a352cc86bd7cf6e9de42894982a2e77d"} Nov 02 14:21:49.363221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:49.363193 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0815eee48859c7200fde6af67f9f1951a352cc86bd7cf6e9de42894982a2e77d" Nov 02 14:21:49.363221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:49.363155 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj" Nov 02 14:21:51.033413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:51.033385 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq"] Nov 02 14:21:51.046169 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:51.046134 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d72354-5f2dq"] Nov 02 14:21:51.077981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:51.077107 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd209d2f-9ec4-41e5-9135-cc0752e384c6" path="/var/lib/kubelet/pods/cd209d2f-9ec4-41e5-9135-cc0752e384c6/volumes" Nov 02 14:21:51.285770 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:21:51.285689 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:21:52.844408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:52.844378 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:21:54.685595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.685558 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4"] Nov 02 14:21:54.697595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.697565 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9fc7d82-zt2w4"] Nov 02 14:21:54.818656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.818615 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd"] Nov 02 14:21:54.818948 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.818919 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" containerID="cri-o://56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49" gracePeriod=30 Nov 02 14:21:54.833668 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.833641 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5"] Nov 02 14:21:54.833892 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.833869 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" containerID="cri-o://1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066" gracePeriod=30 Nov 02 14:21:54.837606 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.837580 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5"] Nov 02 14:21:54.837801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.837756 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" containerID="cri-o://c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a" gracePeriod=30 Nov 02 14:21:54.849874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.849844 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79"] Nov 02 14:21:54.850122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.850079 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" containerID="cri-o://18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1" gracePeriod=30 Nov 02 14:21:54.852471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.852446 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d"] Nov 02 14:21:54.852664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:54.852641 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" containerID="cri-o://bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024" gracePeriod=30 Nov 02 14:21:55.087817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:55.087712 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe" path="/var/lib/kubelet/pods/f8ea4e45-1b6c-4ea1-9b25-7656a9f9f9fe/volumes" Nov 02 14:21:56.434086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.434045 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8"] Nov 02 14:21:56.435625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.435589 4431 cpu_manager.go:414] "containerMap: removing container" podUID="540f1c37-7470-43e5-9f55-6b84b0d5ad18" containerName="revoke-keys" Nov 02 14:21:56.437002 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.436983 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540f1c37-7470-43e5-9f55-6b84b0d5ad18" containerName="revoke-keys" Nov 02 14:21:56.437339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.437323 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="540f1c37-7470-43e5-9f55-6b84b0d5ad18" containerName="revoke-keys" Nov 02 14:21:56.454933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.453707 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.458098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.458073 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8"] Nov 02 14:21:56.476818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.472888 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-87a1a06b-gw-openshift-default-dockercfg-x6m2f\"" Nov 02 14:21:56.531530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.531496 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-certs\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.531689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.531535 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-data\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.531793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.531677 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-credential-socket\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.531793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.531719 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62790564-506b-4f53-a9ab-582b2565bc29-istiod-ca-cert\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.531923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.531802 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktsdr\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-kube-api-access-ktsdr\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.531923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.531840 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-istio-token\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.532043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.531975 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-socket\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.532043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.532020 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62790564-506b-4f53-a9ab-582b2565bc29-istio-podinfo\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.532178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.532154 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-envoy\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.633742 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-certs\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.633840 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-data\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.634136 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-credential-socket\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.634177 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62790564-506b-4f53-a9ab-582b2565bc29-istiod-ca-cert\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.634232 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-certs\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.634250 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktsdr\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-kube-api-access-ktsdr\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.634308 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-istio-token\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.634386 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-socket\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.634432 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62790564-506b-4f53-a9ab-582b2565bc29-istio-podinfo\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.634581 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-envoy\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.635157 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-socket\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.635183 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-credential-socket\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.635800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.635265 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-data\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.636906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.636559 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62790564-506b-4f53-a9ab-582b2565bc29-istiod-ca-cert\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.640796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.637661 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-envoy\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.663287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.663214 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62790564-506b-4f53-a9ab-582b2565bc29-istio-podinfo\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.668697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.668669 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-istio-token\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.693076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.693005 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktsdr\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-kube-api-access-ktsdr\") pod \"e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:56.775801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:56.774947 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:57.181654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:57.181620 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8"] Nov 02 14:21:57.189335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:57.189294 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:21:57.189419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:57.189390 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:21:57.189463 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:57.189430 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:21:57.464564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:57.464468 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" event={"ID":"62790564-506b-4f53-a9ab-582b2565bc29","Type":"ContainerStarted","Data":"9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5"} Nov 02 14:21:57.464564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:57.464529 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" event={"ID":"62790564-506b-4f53-a9ab-582b2565bc29","Type":"ContainerStarted","Data":"5a76aa745cbbec80a4f51dcabb1905e932fc10570b82e3c2adc76592ec206281"} Nov 02 14:21:57.517371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:57.517311 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" podStartSLOduration=1.51729248 podStartE2EDuration="1.51729248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:57.513165388 +0000 UTC m=+1482.550270321" watchObservedRunningTime="2026-11-02 14:21:57.51729248 +0000 UTC m=+1482.554397413" Nov 02 14:21:57.775836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:57.775713 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:58.782484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:58.782454 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:58.989292 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:58.989255 4431 patch_prober.go:29] interesting pod/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 02 14:21:58.989474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:58.989315 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:21:59.525649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:59.525613 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:59.527144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:59.526746 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:21:59.779955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:59.779887 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-a964bc"] Nov 02 14:21:59.782966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:59.782948 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-a964bc" Nov 02 14:21:59.799275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:59.799242 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nz2z6\"" Nov 02 14:21:59.800255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:59.800236 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-a964bc"] Nov 02 14:21:59.913468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:21:59.913437 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgfb9\" (UniqueName: \"kubernetes.io/projected/fce6e3b2-b8b2-4375-8e9c-db65f5156ff6-kube-api-access-bgfb9\") pod \"test-curl-7816-a964bc\" (UID: \"fce6e3b2-b8b2-4375-8e9c-db65f5156ff6\") " pod="opendatahub/test-curl-7816-a964bc" Nov 02 14:22:00.018884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.018273 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgfb9\" (UniqueName: \"kubernetes.io/projected/fce6e3b2-b8b2-4375-8e9c-db65f5156ff6-kube-api-access-bgfb9\") pod \"test-curl-7816-a964bc\" (UID: \"fce6e3b2-b8b2-4375-8e9c-db65f5156ff6\") " pod="opendatahub/test-curl-7816-a964bc" Nov 02 14:22:00.055868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.054105 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj"] Nov 02 14:22:00.060754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.059361 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgfb9\" (UniqueName: \"kubernetes.io/projected/fce6e3b2-b8b2-4375-8e9c-db65f5156ff6-kube-api-access-bgfb9\") pod \"test-curl-7816-a964bc\" (UID: \"fce6e3b2-b8b2-4375-8e9c-db65f5156ff6\") " pod="opendatahub/test-curl-7816-a964bc" Nov 02 14:22:00.060754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.060015 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5fef3f9-cgjqj"] Nov 02 14:22:00.095885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.095810 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-a964bc" Nov 02 14:22:00.176473 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.176442 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221066 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-certs\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221152 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-kube-api-access-4nskf\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-kube-api-access-4nskf\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221222 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-data\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221256 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istiod-ca-cert\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221286 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-credential-socket\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221358 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-podinfo\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221420 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-socket\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221468 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-envoy\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.222069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.221536 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-token\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-token\") pod \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\" (UID: \"d1592382-9355-4bff-aefe-1f4e7b5e4aa8\") " Nov 02 14:22:00.223656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.222930 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-certs" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.223656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.223384 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istiod-ca-cert" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:00.224423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.224168 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-socket" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.225626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.225391 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-data" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.225626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.225539 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.225626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.225590 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.225626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.225607 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.226453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.226431 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-credential-socket" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.244308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.244264 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-token" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.246100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.246063 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-kube-api-access-4nskf" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "kube-api-access-4nskf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.252274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.252237 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-envoy" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.253278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.253239 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-podinfo" pod "d1592382-9355-4bff-aefe-1f4e7b5e4aa8" (UID: "d1592382-9355-4bff-aefe-1f4e7b5e4aa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:22:00.269950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.269849 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:22:00.275938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.275880 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:22:00.277642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.277584 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:22:00.289093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.289023 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:22:00.326167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326130 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-token\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-token\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.326342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326195 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8ff025-8014-4768-b7d3-37c10d3471f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8ff025-8014-4768-b7d3-37c10d3471f8-istiod-ca-cert\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.326342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326277 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-credential-socket\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.326469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326392 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-certs\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326561 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59160f47-0e62-4d4a-97e3-dd608ffabf72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59160f47-0e62-4d4a-97e3-dd608ffabf72-istiod-ca-cert\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326624 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-socket\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326656 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-envoy\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326682 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-envoy\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326725 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-data\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326766 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-podinfo\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326846 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-kube-api-access-cwpp7\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-kube-api-access-cwpp7\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326883 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-token\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-token\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326912 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-data\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326939 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-podinfo\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.326988 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-kube-api-access-xlj7s\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-kube-api-access-xlj7s\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.327025 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-envoy\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.327072 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-credential-socket\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.327124 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-kube-api-access-zvtnb\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-kube-api-access-zvtnb\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.327152 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-socket\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.327191 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-credential-socket\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.327260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.327223 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-socket\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.328801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.328312 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istiod-ca-cert\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.328801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.328355 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-data\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.328801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.328414 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-token\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-token\") pod \"59160f47-0e62-4d4a-97e3-dd608ffabf72\" (UID: \"59160f47-0e62-4d4a-97e3-dd608ffabf72\") " Nov 02 14:22:00.328801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.328487 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-certs\") pod \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\" (UID: \"f7e3f5a6-db61-4edf-b6d7-e2839a55d659\") " Nov 02 14:22:00.328801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.328518 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-podinfo\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.328801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.328575 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-certs\") pod \"2d8ff025-8014-4768-b7d3-37c10d3471f8\" (UID: \"2d8ff025-8014-4768-b7d3-37c10d3471f8\") " Nov 02 14:22:00.329876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.329698 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.329876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.329724 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.329876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.329739 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nskf\" (UniqueName: \"kubernetes.io/projected/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-kube-api-access-4nskf\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.329876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.329752 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.330189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.329878 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8ff025-8014-4768-b7d3-37c10d3471f8-istiod-ca-cert" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:00.330375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.330270 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.330375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.330302 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1592382-9355-4bff-aefe-1f4e7b5e4aa8-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.331015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.330728 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-certs" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.331015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.330883 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-podinfo" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:22:00.331213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.331095 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-socket" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.331555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.331378 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59160f47-0e62-4d4a-97e3-dd608ffabf72-istiod-ca-cert" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:00.331689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.331669 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-credential-socket" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.331893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.331861 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-certs" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.333452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.332601 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-credential-socket" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.333452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.332907 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-socket" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.333452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.333230 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istiod-ca-cert" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:00.333662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.333593 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-certs" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.333922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.333850 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-socket" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.334043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.333912 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-data" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.334108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.334077 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-kube-api-access-cwpp7" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "kube-api-access-cwpp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.334338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.334314 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-data" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.334553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.334535 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-credential-socket" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.335202 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.334955 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-data" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.336024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.335996 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-token" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.337384 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.337360 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-podinfo" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:22:00.337494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.337475 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-kube-api-access-zvtnb" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "kube-api-access-zvtnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.337705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.337684 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-kube-api-access-xlj7s" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "kube-api-access-xlj7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.338471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.338450 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-podinfo" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:22:00.339463 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.339442 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-envoy" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.339576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.339512 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-envoy" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.341263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.341228 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-envoy" pod "59160f47-0e62-4d4a-97e3-dd608ffabf72" (UID: "59160f47-0e62-4d4a-97e3-dd608ffabf72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.342411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.342384 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-token" pod "f7e3f5a6-db61-4edf-b6d7-e2839a55d659" (UID: "f7e3f5a6-db61-4edf-b6d7-e2839a55d659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.343925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.343890 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-token" pod "2d8ff025-8014-4768-b7d3-37c10d3471f8" (UID: "2d8ff025-8014-4768-b7d3-37c10d3471f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.431723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.431694 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cccf69b9-f541-4987-b45a-649d076d545a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccf69b9-f541-4987-b45a-649d076d545a-istiod-ca-cert\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.431905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.431738 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-credential-socket\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.431905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.431791 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-socket\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.431905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.431818 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-certs\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.432060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.431924 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-kube-api-access-78qvn\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-kube-api-access-78qvn\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.432060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432025 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cccf69b9-f541-4987-b45a-649d076d545a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccf69b9-f541-4987-b45a-649d076d545a-istio-podinfo\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.432176 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432067 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-credential-socket" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.432176 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432132 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-socket" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.432307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432218 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccf69b9-f541-4987-b45a-649d076d545a-istiod-ca-cert" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:00.432307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432216 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-certs" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.432387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432320 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-data\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.432431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432416 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-istio-token\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-istio-token\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.432529 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432511 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-envoy\") pod \"cccf69b9-f541-4987-b45a-649d076d545a\" (UID: \"cccf69b9-f541-4987-b45a-649d076d545a\") " Nov 02 14:22:00.432529 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.432519 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-data" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.433110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433067 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59160f47-0e62-4d4a-97e3-dd608ffabf72-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433229 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433211 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433230 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433246 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433261 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433274 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433287 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cccf69b9-f541-4987-b45a-649d076d545a-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433300 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433316 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpp7\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-kube-api-access-cwpp7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433333 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433347 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433359 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433371 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433383 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433394 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlj7s\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-kube-api-access-xlj7s\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433407 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433420 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433434 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvtnb\" (UniqueName: \"kubernetes.io/projected/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-kube-api-access-zvtnb\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433449 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433463 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433478 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433491 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433504 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433517 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59160f47-0e62-4d4a-97e3-dd608ffabf72-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433529 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e3f5a6-db61-4edf-b6d7-e2839a55d659-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433543 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433555 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8ff025-8014-4768-b7d3-37c10d3471f8-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433567 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8ff025-8014-4768-b7d3-37c10d3471f8-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433579 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8ff025-8014-4768-b7d3-37c10d3471f8-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433594 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433606 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.433845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.433619 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59160f47-0e62-4d4a-97e3-dd608ffabf72-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.435017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.434732 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-kube-api-access-78qvn" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "kube-api-access-78qvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.435017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.434746 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cccf69b9-f541-4987-b45a-649d076d545a-istio-podinfo" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:22:00.435017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.434947 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-envoy" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:00.436394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.436370 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-istio-token" pod "cccf69b9-f541-4987-b45a-649d076d545a" (UID: "cccf69b9-f541-4987-b45a-649d076d545a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:00.461024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.460997 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-a964bc"] Nov 02 14:22:00.524147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.524114 4431 generic.go:356] "Generic (PLEG): container finished" podID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerID="1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066" exitCode=0 Nov 02 14:22:00.524346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.524312 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" event={"ID":"d1592382-9355-4bff-aefe-1f4e7b5e4aa8","Type":"ContainerDied","Data":"1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066"} Nov 02 14:22:00.524510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.524383 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" event={"ID":"d1592382-9355-4bff-aefe-1f4e7b5e4aa8","Type":"ContainerDied","Data":"db8ca96312f695e32fed568797deca466ed9842081b47f13634d072ba48db4ae"} Nov 02 14:22:00.524510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.524420 4431 scope.go:122] "RemoveContainer" containerID="1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066" Nov 02 14:22:00.524818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.524583 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5" Nov 02 14:22:00.527291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.527267 4431 generic.go:356] "Generic (PLEG): container finished" podID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerID="18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1" exitCode=0 Nov 02 14:22:00.527440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.527339 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" event={"ID":"f7e3f5a6-db61-4edf-b6d7-e2839a55d659","Type":"ContainerDied","Data":"18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1"} Nov 02 14:22:00.527440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.527377 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" event={"ID":"f7e3f5a6-db61-4edf-b6d7-e2839a55d659","Type":"ContainerDied","Data":"b0a51c14538eac856abb7f3fb2f327162afe57a574512b702dec4f4035f93038"} Nov 02 14:22:00.527566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.527448 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79" Nov 02 14:22:00.530355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.530328 4431 generic.go:356] "Generic (PLEG): container finished" podID="cccf69b9-f541-4987-b45a-649d076d545a" containerID="c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a" exitCode=0 Nov 02 14:22:00.530484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.530468 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" event={"ID":"cccf69b9-f541-4987-b45a-649d076d545a","Type":"ContainerDied","Data":"c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a"} Nov 02 14:22:00.530816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.530509 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" event={"ID":"cccf69b9-f541-4987-b45a-649d076d545a","Type":"ContainerDied","Data":"e6a882b978621b6a02340e0e86a45f40c16f7f5241ed67515c899d6eb99dfb73"} Nov 02 14:22:00.530816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.530581 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5" Nov 02 14:22:00.533081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.533047 4431 generic.go:356] "Generic (PLEG): container finished" podID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerID="56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49" exitCode=0 Nov 02 14:22:00.533170 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.533139 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" event={"ID":"2d8ff025-8014-4768-b7d3-37c10d3471f8","Type":"ContainerDied","Data":"56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49"} Nov 02 14:22:00.533246 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.533179 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" event={"ID":"2d8ff025-8014-4768-b7d3-37c10d3471f8","Type":"ContainerDied","Data":"c631b008b88107e68e9517d9aee8f773861e6741ef4ad9fda0c1e8a5af963f33"} Nov 02 14:22:00.533308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.533252 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd" Nov 02 14:22:00.535076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.535051 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78qvn\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-kube-api-access-78qvn\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.535217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.535202 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cccf69b9-f541-4987-b45a-649d076d545a-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.535302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.535291 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cccf69b9-f541-4987-b45a-649d076d545a-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.535389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.535378 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cccf69b9-f541-4987-b45a-649d076d545a-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:00.539089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.538628 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-a964bc" event={"ID":"fce6e3b2-b8b2-4375-8e9c-db65f5156ff6","Type":"ContainerStarted","Data":"e8959816861eee38b46fe2dbf7596e238110cfa687bf3ed715adc5e738357949"} Nov 02 14:22:00.540933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.540832 4431 generic.go:356] "Generic (PLEG): container finished" podID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerID="bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024" exitCode=0 Nov 02 14:22:00.542590 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.541862 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" Nov 02 14:22:00.543144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.543046 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" event={"ID":"59160f47-0e62-4d4a-97e3-dd608ffabf72","Type":"ContainerDied","Data":"bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024"} Nov 02 14:22:00.543144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.543094 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d" event={"ID":"59160f47-0e62-4d4a-97e3-dd608ffabf72","Type":"ContainerDied","Data":"64aff65fa698312279e0eefb33b8c932537b9c4b1b3859fbf127dd7564985104"} Nov 02 14:22:00.545315 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.545295 4431 scope.go:122] "RemoveContainer" containerID="1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066" Nov 02 14:22:00.546500 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:00.546471 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066\": container with ID starting with 1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066 not found: ID does not exist" containerID="1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066" Nov 02 14:22:00.546617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.546505 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066"} err="failed to get container status \"1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066\": rpc error: code = NotFound desc = could not find container \"1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066\": container with ID starting with 1abdae091470cad23e41fd6b49394146bf40431ae4b0c691ed579a1769e34066 not found: ID does not exist" Nov 02 14:22:00.546617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.546529 4431 scope.go:122] "RemoveContainer" containerID="18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1" Nov 02 14:22:00.560662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.560642 4431 scope.go:122] "RemoveContainer" containerID="18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1" Nov 02 14:22:00.561051 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:00.561020 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1\": container with ID starting with 18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1 not found: ID does not exist" containerID="18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1" Nov 02 14:22:00.562148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.561057 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1"} err="failed to get container status \"18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1\": rpc error: code = NotFound desc = could not find container \"18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1\": container with ID starting with 18a7c40b4bc105a4d67aa1afb3c6a74614d9829dcc72c38e434113203ae482c1 not found: ID does not exist" Nov 02 14:22:00.562148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.561086 4431 scope.go:122] "RemoveContainer" containerID="c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a" Nov 02 14:22:00.571635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.571618 4431 scope.go:122] "RemoveContainer" containerID="c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a" Nov 02 14:22:00.572030 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:00.572006 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a\": container with ID starting with c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a not found: ID does not exist" containerID="c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a" Nov 02 14:22:00.572142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.572036 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a"} err="failed to get container status \"c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a\": rpc error: code = NotFound desc = could not find container \"c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a\": container with ID starting with c718ab77f3856e5ad472bda22ae6ac6e441178f4a0b34a67f8bc313330a4dc7a not found: ID does not exist" Nov 02 14:22:00.572142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.572053 4431 scope.go:122] "RemoveContainer" containerID="56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49" Nov 02 14:22:00.586183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.586164 4431 scope.go:122] "RemoveContainer" containerID="90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb" Nov 02 14:22:00.595401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.595383 4431 scope.go:122] "RemoveContainer" containerID="56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49" Nov 02 14:22:00.595643 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:00.595626 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49\": container with ID starting with 56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49 not found: ID does not exist" containerID="56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49" Nov 02 14:22:00.595684 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.595652 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49"} err="failed to get container status \"56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49\": rpc error: code = NotFound desc = could not find container \"56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49\": container with ID starting with 56b8d3e648aaae2d2a43c04bae4fff4fdcaad12b0cd46b34cd75f72f5e180f49 not found: ID does not exist" Nov 02 14:22:00.595684 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.595669 4431 scope.go:122] "RemoveContainer" containerID="90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb" Nov 02 14:22:00.595901 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:00.595885 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb\": container with ID starting with 90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb not found: ID does not exist" containerID="90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb" Nov 02 14:22:00.595944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.595905 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb"} err="failed to get container status \"90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb\": rpc error: code = NotFound desc = could not find container \"90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb\": container with ID starting with 90ae9adcb5600db8c7ce72fa7075f87c83f687505bb39272f000e79a72471ddb not found: ID does not exist" Nov 02 14:22:00.595944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.595919 4431 scope.go:122] "RemoveContainer" containerID="bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024" Nov 02 14:22:00.604821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.604770 4431 scope.go:122] "RemoveContainer" containerID="bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024" Nov 02 14:22:00.605079 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:00.605061 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024\": container with ID starting with bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024 not found: ID does not exist" containerID="bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024" Nov 02 14:22:00.605123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.605089 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024"} err="failed to get container status \"bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024\": rpc error: code = NotFound desc = could not find container \"bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024\": container with ID starting with bc9f29f8dfe45fb3e5fe5fd1add01c4e6caa7913d4f67c0b9f27745ea16e9024 not found: ID does not exist" Nov 02 14:22:00.660762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.660732 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5"] Nov 02 14:22:00.664440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.664417 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ngbv5"] Nov 02 14:22:00.700083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.700058 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd"] Nov 02 14:22:00.706344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.706323 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c9fc7d82-openshift-default-64bfc7885c-sj7kd"] Nov 02 14:22:00.743289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.743265 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79"] Nov 02 14:22:00.747179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.747157 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-zbf79"] Nov 02 14:22:00.784646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.784618 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5"] Nov 02 14:22:00.790871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.790844 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-nfst5"] Nov 02 14:22:00.825328 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.825270 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d"] Nov 02 14:22:00.834308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:00.834274 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4be645-openshift-default-ccc9d9d8f-ntc7d"] Nov 02 14:22:01.081001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.078914 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098df760-9588-4719-9283-6dec0c8676e4" path="/var/lib/kubelet/pods/098df760-9588-4719-9283-6dec0c8676e4/volumes" Nov 02 14:22:01.081001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.079652 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" path="/var/lib/kubelet/pods/2d8ff025-8014-4768-b7d3-37c10d3471f8/volumes" Nov 02 14:22:01.081001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.080355 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" path="/var/lib/kubelet/pods/59160f47-0e62-4d4a-97e3-dd608ffabf72/volumes" Nov 02 14:22:01.083623 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.081759 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cccf69b9-f541-4987-b45a-649d076d545a" path="/var/lib/kubelet/pods/cccf69b9-f541-4987-b45a-649d076d545a/volumes" Nov 02 14:22:01.083623 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.082449 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" path="/var/lib/kubelet/pods/d1592382-9355-4bff-aefe-1f4e7b5e4aa8/volumes" Nov 02 14:22:01.083623 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.083122 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" path="/var/lib/kubelet/pods/f7e3f5a6-db61-4edf-b6d7-e2839a55d659/volumes" Nov 02 14:22:01.553389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.553346 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-a964bc" event={"ID":"fce6e3b2-b8b2-4375-8e9c-db65f5156ff6","Type":"ContainerStarted","Data":"cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427"} Nov 02 14:22:01.562669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.562644 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth"] Nov 02 14:22:01.563254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563231 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" Nov 02 14:22:01.563254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563247 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" Nov 02 14:22:01.563385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563265 4431 cpu_manager.go:414] "containerMap: removing container" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" Nov 02 14:22:01.563385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563271 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" Nov 02 14:22:01.563385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563282 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" Nov 02 14:22:01.563385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563287 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" Nov 02 14:22:01.563385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563305 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" Nov 02 14:22:01.563385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563309 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" Nov 02 14:22:01.563385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563329 4431 cpu_manager.go:414] "containerMap: removing container" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" Nov 02 14:22:01.563385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563334 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" Nov 02 14:22:01.563659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563416 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1592382-9355-4bff-aefe-1f4e7b5e4aa8" containerName="istio-proxy" Nov 02 14:22:01.563659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563427 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="59160f47-0e62-4d4a-97e3-dd608ffabf72" containerName="istio-proxy" Nov 02 14:22:01.563659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563435 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" Nov 02 14:22:01.563659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563443 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e3f5a6-db61-4edf-b6d7-e2839a55d659" containerName="istio-proxy" Nov 02 14:22:01.563659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563450 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" Nov 02 14:22:01.563659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.563457 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cccf69b9-f541-4987-b45a-649d076d545a" containerName="istio-proxy" Nov 02 14:22:01.566825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.566809 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:01.585585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.585558 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-87a1a06b\"" Nov 02 14:22:01.589881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.589855 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth"] Nov 02 14:22:01.590011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.589998 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-87a1a06b-dockercfg-bvqc8\"" Nov 02 14:22:01.608368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.608317 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7816-a964bc" podStartSLOduration=1.7026560530000001 podStartE2EDuration="2.608305169s" totalImagesPullingTime="905.649116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:01.60633053 +0000 UTC m=+1486.643435462" watchObservedRunningTime="2026-11-02 14:22:01.608305169 +0000 UTC m=+1486.645410100" Nov 02 14:22:01.647937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.647891 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478cc5a8-20dd-4e7d-b48c-214fab27e94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth\" (UID: \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:01.648538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.648259 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fz2s\" (UniqueName: \"kubernetes.io/projected/478cc5a8-20dd-4e7d-b48c-214fab27e94c-kube-api-access-6fz2s\") pod \"payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth\" (UID: \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:01.654786 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.654735 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95"] Nov 02 14:22:01.658261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.655657 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" Nov 02 14:22:01.658261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.655675 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8ff025-8014-4768-b7d3-37c10d3471f8" containerName="istio-proxy" Nov 02 14:22:01.759980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.759939 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478cc5a8-20dd-4e7d-b48c-214fab27e94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth\" (UID: \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:01.760307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.760285 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fz2s\" (UniqueName: \"kubernetes.io/projected/478cc5a8-20dd-4e7d-b48c-214fab27e94c-kube-api-access-6fz2s\") pod \"payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth\" (UID: \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:01.762279 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.762255 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:01.765725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.765405 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478cc5a8-20dd-4e7d-b48c-214fab27e94c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth\" (UID: \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:01.773198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.773173 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95"] Nov 02 14:22:01.792135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.792102 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fz2s\" (UniqueName: \"kubernetes.io/projected/478cc5a8-20dd-4e7d-b48c-214fab27e94c-kube-api-access-6fz2s\") pod \"payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth\" (UID: \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:01.862062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.862033 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049be275-7779-40a3-8fc8-bdadb5ffad3c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95\" (UID: \"049be275-7779-40a3-8fc8-bdadb5ffad3c\") " pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:01.862226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.862069 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48r77\" (UniqueName: \"kubernetes.io/projected/049be275-7779-40a3-8fc8-bdadb5ffad3c-kube-api-access-48r77\") pod \"payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95\" (UID: \"049be275-7779-40a3-8fc8-bdadb5ffad3c\") " pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:01.881099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.881070 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:01.963715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.963680 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049be275-7779-40a3-8fc8-bdadb5ffad3c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95\" (UID: \"049be275-7779-40a3-8fc8-bdadb5ffad3c\") " pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:01.963897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.963730 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48r77\" (UniqueName: \"kubernetes.io/projected/049be275-7779-40a3-8fc8-bdadb5ffad3c-kube-api-access-48r77\") pod \"payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95\" (UID: \"049be275-7779-40a3-8fc8-bdadb5ffad3c\") " pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:01.964721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.964689 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049be275-7779-40a3-8fc8-bdadb5ffad3c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95\" (UID: \"049be275-7779-40a3-8fc8-bdadb5ffad3c\") " pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:01.989015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:01.988981 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48r77\" (UniqueName: \"kubernetes.io/projected/049be275-7779-40a3-8fc8-bdadb5ffad3c-kube-api-access-48r77\") pod \"payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95\" (UID: \"049be275-7779-40a3-8fc8-bdadb5ffad3c\") " pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:02.090490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.090457 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:02.189970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.189938 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth"] Nov 02 14:22:02.397849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.397748 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95"] Nov 02 14:22:02.398642 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:02.398616 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049be275_7779_40a3_8fc8_bdadb5ffad3c.slice/crio-55d1c7ca318684a82ea02aed39e1efd3ef72f6c5933e72bfb98ce3388f109997 WatchSource:0}: Error finding container 55d1c7ca318684a82ea02aed39e1efd3ef72f6c5933e72bfb98ce3388f109997: Status 404 returned error can't find the container with id 55d1c7ca318684a82ea02aed39e1efd3ef72f6c5933e72bfb98ce3388f109997 Nov 02 14:22:02.517336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.517305 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-a964bc"] Nov 02 14:22:02.522209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.522178 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-a964bc"] Nov 02 14:22:02.564714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.564662 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" event={"ID":"049be275-7779-40a3-8fc8-bdadb5ffad3c","Type":"ContainerStarted","Data":"a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79"} Nov 02 14:22:02.565104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.564738 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" event={"ID":"049be275-7779-40a3-8fc8-bdadb5ffad3c","Type":"ContainerStarted","Data":"55d1c7ca318684a82ea02aed39e1efd3ef72f6c5933e72bfb98ce3388f109997"} Nov 02 14:22:02.565104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.565040 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:02.566880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.566850 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" event={"ID":"478cc5a8-20dd-4e7d-b48c-214fab27e94c","Type":"ContainerStarted","Data":"377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9"} Nov 02 14:22:02.566981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.566906 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" event={"ID":"478cc5a8-20dd-4e7d-b48c-214fab27e94c","Type":"ContainerStarted","Data":"6a00135c46fe61eb1bc4629fb2652fb156ba9c32b31aacde2cae7d78287ca5e2"} Nov 02 14:22:02.610453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.610389 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" podStartSLOduration=1.6103691740000001 podStartE2EDuration="1.610369174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:02.607764895 +0000 UTC m=+1487.644869829" watchObservedRunningTime="2026-11-02 14:22:02.610369174 +0000 UTC m=+1487.647474108" Nov 02 14:22:02.653062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.653000 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" podStartSLOduration=1.652985269 podStartE2EDuration="1.652985269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:02.651750508 +0000 UTC m=+1487.688855461" watchObservedRunningTime="2026-11-02 14:22:02.652985269 +0000 UTC m=+1487.690090201" Nov 02 14:22:02.748770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.748690 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9"] Nov 02 14:22:02.749286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.749273 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fce6e3b2-b8b2-4375-8e9c-db65f5156ff6" containerName="test-curl-7816-a964bc" Nov 02 14:22:02.749327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.749287 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce6e3b2-b8b2-4375-8e9c-db65f5156ff6" containerName="test-curl-7816-a964bc" Nov 02 14:22:02.749381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.749371 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce6e3b2-b8b2-4375-8e9c-db65f5156ff6" containerName="test-curl-7816-a964bc" Nov 02 14:22:02.752890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.752874 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:02.775525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.775489 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:22:02.775837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.775562 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:22:02.781509 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.781476 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9"] Nov 02 14:22:02.878016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.877985 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:02.878508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.878154 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:02.878508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.878281 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79bh\" (UniqueName: \"kubernetes.io/projected/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-kube-api-access-v79bh\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:02.979221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.979189 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:02.979385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.979320 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:02.979446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.979423 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79bh\" (UniqueName: \"kubernetes.io/projected/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-kube-api-access-v79bh\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:02.980134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.980092 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:02.981926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:02.981905 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:03.004186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:03.004121 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79bh\" (UniqueName: \"kubernetes.io/projected/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-kube-api-access-v79bh\") pod \"maas-api-revoke-sub-211b8cf5-cbgj9\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:03.080351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:03.080315 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:03.388687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:03.388659 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9"] Nov 02 14:22:03.389892 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:03.389867 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1073b3b8_d6e4_4ef7_a1fd_2c343e47af64.slice/crio-51294079cae4604de496e095237c7015d502df81b2e2988892746975f212d1e9 WatchSource:0}: Error finding container 51294079cae4604de496e095237c7015d502df81b2e2988892746975f212d1e9: Status 404 returned error can't find the container with id 51294079cae4604de496e095237c7015d502df81b2e2988892746975f212d1e9 Nov 02 14:22:03.575710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:03.575675 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" event={"ID":"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64","Type":"ContainerStarted","Data":"51294079cae4604de496e095237c7015d502df81b2e2988892746975f212d1e9"} Nov 02 14:22:03.575899 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:03.575872 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7816-a964bc" podUID="fce6e3b2-b8b2-4375-8e9c-db65f5156ff6" containerName="test-curl-7816-a964bc" containerID="cri-o://cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427" gracePeriod=2 Nov 02 14:22:03.576276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:03.576197 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:04.584681 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:04.584596 4431 generic.go:356] "Generic (PLEG): container finished" podID="1073b3b8-d6e4-4ef7-a1fd-2c343e47af64" containerID="4fd1f899050d0eec9a8b400fc26864b872728c34877ccbfd3c09429051aafb62" exitCode=0 Nov 02 14:22:04.585150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:04.584679 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" event={"ID":"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64","Type":"ContainerDied","Data":"4fd1f899050d0eec9a8b400fc26864b872728c34877ccbfd3c09429051aafb62"} Nov 02 14:22:05.110593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.110560 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx"] Nov 02 14:22:05.114553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.114533 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:05.133819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.133793 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-c05168\"/\"openshift-service-ca.crt\"" Nov 02 14:22:05.133943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.133821 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-c05168\"/\"kube-root-ca.crt\"" Nov 02 14:22:05.136469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.136447 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx"] Nov 02 14:22:05.146114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.146093 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-c05168\"/\"e2e-usage-sim-c8b392-cert\"" Nov 02 14:22:05.146228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.146115 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-c05168\"/\"default-dockercfg-8cs8z\"" Nov 02 14:22:05.202631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.202555 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvps\" (UniqueName: \"kubernetes.io/projected/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-kube-api-access-xvvps\") pod \"e2e-usage-sim-c8b392-54bf74765c-5dhbx\" (UID: \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\") " pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:05.202844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.202676 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-tls\") pod \"e2e-usage-sim-c8b392-54bf74765c-5dhbx\" (UID: \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\") " pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:05.303553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.303517 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-tls\") pod \"e2e-usage-sim-c8b392-54bf74765c-5dhbx\" (UID: \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\") " pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:05.303752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.303696 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvps\" (UniqueName: \"kubernetes.io/projected/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-kube-api-access-xvvps\") pod \"e2e-usage-sim-c8b392-54bf74765c-5dhbx\" (UID: \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\") " pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:05.306110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.306087 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-tls\") pod \"e2e-usage-sim-c8b392-54bf74765c-5dhbx\" (UID: \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\") " pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:05.328422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.328397 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvps\" (UniqueName: \"kubernetes.io/projected/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-kube-api-access-xvvps\") pod \"e2e-usage-sim-c8b392-54bf74765c-5dhbx\" (UID: \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\") " pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:05.427111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.427089 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:05.733172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.733145 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:05.749416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.749389 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx"] Nov 02 14:22:05.749848 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:05.749822 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a0010ea_8eef_4b5f_b1fc_b85d49aa5607.slice/crio-f8eb112529a91498f69fe93d02592fdaf4a7a6d49120660408bc510e692132da WatchSource:0}: Error finding container f8eb112529a91498f69fe93d02592fdaf4a7a6d49120660408bc510e692132da: Status 404 returned error can't find the container with id f8eb112529a91498f69fe93d02592fdaf4a7a6d49120660408bc510e692132da Nov 02 14:22:05.784061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.784035 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-a964bc" Nov 02 14:22:05.810515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.810485 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-kube-api-access-v79bh\" (UniqueName: \"kubernetes.io/projected/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-kube-api-access-v79bh\") pod \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " Nov 02 14:22:05.810755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.810733 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-xks-ca\") pod \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " Nov 02 14:22:05.810869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.810817 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-openshift-ca\") pod \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\" (UID: \"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64\") " Nov 02 14:22:05.810931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.810916 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce6e3b2-b8b2-4375-8e9c-db65f5156ff6-kube-api-access-bgfb9\" (UniqueName: \"kubernetes.io/projected/fce6e3b2-b8b2-4375-8e9c-db65f5156ff6-kube-api-access-bgfb9\") pod \"fce6e3b2-b8b2-4375-8e9c-db65f5156ff6\" (UID: \"fce6e3b2-b8b2-4375-8e9c-db65f5156ff6\") " Nov 02 14:22:05.811382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.811354 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-openshift-ca" pod "1073b3b8-d6e4-4ef7-a1fd-2c343e47af64" (UID: "1073b3b8-d6e4-4ef7-a1fd-2c343e47af64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:05.814113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.814069 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-xks-ca" pod "1073b3b8-d6e4-4ef7-a1fd-2c343e47af64" (UID: "1073b3b8-d6e4-4ef7-a1fd-2c343e47af64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:22:05.814268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.814149 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce6e3b2-b8b2-4375-8e9c-db65f5156ff6-kube-api-access-bgfb9" pod "fce6e3b2-b8b2-4375-8e9c-db65f5156ff6" (UID: "fce6e3b2-b8b2-4375-8e9c-db65f5156ff6"). InnerVolumeSpecName "kube-api-access-bgfb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:05.814268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.814203 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-kube-api-access-v79bh" pod "1073b3b8-d6e4-4ef7-a1fd-2c343e47af64" (UID: "1073b3b8-d6e4-4ef7-a1fd-2c343e47af64"). InnerVolumeSpecName "kube-api-access-v79bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:05.912468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.912440 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:05.912468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.912465 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:05.912468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.912476 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgfb9\" (UniqueName: \"kubernetes.io/projected/fce6e3b2-b8b2-4375-8e9c-db65f5156ff6-kube-api-access-bgfb9\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:05.912680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:05.912486 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79bh\" (UniqueName: \"kubernetes.io/projected/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64-kube-api-access-v79bh\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:06.602336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.602246 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" event={"ID":"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607","Type":"ContainerStarted","Data":"6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe"} Nov 02 14:22:06.602336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.602308 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:06.602336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.602324 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" event={"ID":"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607","Type":"ContainerStarted","Data":"f8eb112529a91498f69fe93d02592fdaf4a7a6d49120660408bc510e692132da"} Nov 02 14:22:06.603857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.603829 4431 generic.go:356] "Generic (PLEG): container finished" podID="fce6e3b2-b8b2-4375-8e9c-db65f5156ff6" containerID="cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427" exitCode=137 Nov 02 14:22:06.603977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.603879 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-a964bc" Nov 02 14:22:06.603977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.603932 4431 scope.go:122] "RemoveContainer" containerID="cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427" Nov 02 14:22:06.605851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.605830 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" event={"ID":"1073b3b8-d6e4-4ef7-a1fd-2c343e47af64","Type":"ContainerDied","Data":"51294079cae4604de496e095237c7015d502df81b2e2988892746975f212d1e9"} Nov 02 14:22:06.605960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.605875 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51294079cae4604de496e095237c7015d502df81b2e2988892746975f212d1e9" Nov 02 14:22:06.605960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.605925 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9" Nov 02 14:22:06.612203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.612176 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:22:06.614868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.614849 4431 scope.go:122] "RemoveContainer" containerID="cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427" Nov 02 14:22:06.615140 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:06.615122 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427\": container with ID starting with cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427 not found: ID does not exist" containerID="cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427" Nov 02 14:22:06.615209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.615154 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427"} err="failed to get container status \"cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427\": rpc error: code = NotFound desc = could not find container \"cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427\": container with ID starting with cea575c0a34885a364213dd211eb5b95e6e6e58d0790611801cb0edeb352a427 not found: ID does not exist" Nov 02 14:22:06.645948 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:06.645904 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" podStartSLOduration=1.645892739 podStartE2EDuration="1.645892739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:06.643889133 +0000 UTC m=+1491.680994075" watchObservedRunningTime="2026-11-02 14:22:06.645892739 +0000 UTC m=+1491.682997673" Nov 02 14:22:06.658372 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:06.658348 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7816-a964bc\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="fce6e3b2-b8b2-4375-8e9c-db65f5156ff6" pod="opendatahub/test-curl-7816-a964bc" Nov 02 14:22:07.076792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:07.076740 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce6e3b2-b8b2-4375-8e9c-db65f5156ff6" path="/var/lib/kubelet/pods/fce6e3b2-b8b2-4375-8e9c-db65f5156ff6/volumes" Nov 02 14:22:09.435043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.435012 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p"] Nov 02 14:22:09.435740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.435714 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1073b3b8-d6e4-4ef7-a1fd-2c343e47af64" containerName="revoke-keys" Nov 02 14:22:09.435740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.435738 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1073b3b8-d6e4-4ef7-a1fd-2c343e47af64" containerName="revoke-keys" Nov 02 14:22:09.435933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.435917 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1073b3b8-d6e4-4ef7-a1fd-2c343e47af64" containerName="revoke-keys" Nov 02 14:22:09.440624 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.440606 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.458842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.458821 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1a9273a49ced\"" Nov 02 14:22:09.458842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.458828 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 14:22:09.461589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.461559 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-routing-overlay-volume\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.461693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.461675 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-provider-credentials\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.461749 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.461737 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kssx2\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-kube-api-access-kssx2\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.461939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.461924 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-plugins-config-volume\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.462239 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.462221 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p"] Nov 02 14:22:09.472072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.472050 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gnrdp\"" Nov 02 14:22:09.472185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.472071 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 14:22:09.472245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.472184 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 14:22:09.559306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.559267 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 14:22:09.581811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.564612 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kssx2\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-kube-api-access-kssx2\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.581811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.564732 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-plugins-config-volume\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.581811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.565798 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-plugins-config-volume\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.581811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.566205 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-routing-overlay-volume\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.581811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.566280 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-provider-credentials\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.581811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.567506 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-routing-overlay-volume\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.581811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.569642 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-provider-credentials\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.593822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.593583 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kssx2\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-kube-api-access-kssx2\") pod \"payload-processing-external-model-85d6f9ccbb-86w9p\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:09.754134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:09.754059 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:10.070386 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:10.070356 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d97b73d_9190_44cb_b11a_be2421ca0759.slice/crio-daeced38d2b6dcb915c4d47ca5866646fc812afaebc558002f15eae1b0c79796 WatchSource:0}: Error finding container daeced38d2b6dcb915c4d47ca5866646fc812afaebc558002f15eae1b0c79796: Status 404 returned error can't find the container with id daeced38d2b6dcb915c4d47ca5866646fc812afaebc558002f15eae1b0c79796 Nov 02 14:22:10.070815 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:10.070790 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p"] Nov 02 14:22:10.644637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:10.644600 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" event={"ID":"1d97b73d-9190-44cb-b11a-be2421ca0759","Type":"ContainerStarted","Data":"b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9"} Nov 02 14:22:10.645126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:10.644668 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:10.645126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:10.644695 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" event={"ID":"1d97b73d-9190-44cb-b11a-be2421ca0759","Type":"ContainerStarted","Data":"daeced38d2b6dcb915c4d47ca5866646fc812afaebc558002f15eae1b0c79796"} Nov 02 14:22:10.694807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:10.694747 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" podStartSLOduration=1.694729897 podStartE2EDuration="1.694729897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:10.692365264 +0000 UTC m=+1495.729470196" watchObservedRunningTime="2026-11-02 14:22:10.694729897 +0000 UTC m=+1495.731834830" Nov 02 14:22:12.879923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.879880 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr"] Nov 02 14:22:12.885044 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.885008 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:12.909393 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.909363 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmtft\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-kube-api-access-lmtft\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:12.909566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.909405 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:12.909566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.909441 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:12.909880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.909851 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:12.922190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922150 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr"] Nov 02 14:22:12.922332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922312 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-d6506fd29d97\"" Nov 02 14:22:12.922450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922432 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-e2e--d8f5123f-dockercfg-qlpzh\"" Nov 02 14:22:12.922528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922509 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-598f75ee016d\"" Nov 02 14:22:12.922608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922595 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-b8b0c81894ae\"" Nov 02 14:22:12.922652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922642 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"openshift-service-ca.crt\"" Nov 02 14:22:12.922685 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922678 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9ba664\"" Nov 02 14:22:12.922886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922869 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"routing-overlay\"" Nov 02 14:22:12.922954 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922940 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"kube-root-ca.crt\"" Nov 02 14:22:12.922996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:12.922985 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9ba664\"/\"agc-cred-9b4d3424ae3d\"" Nov 02 14:22:13.010922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.010886 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.011089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.010946 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.011187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.011171 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.011262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.011237 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmtft\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-kube-api-access-lmtft\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.011722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.011696 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.011878 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.011701 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.013524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.013493 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.039459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.039432 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmtft\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-kube-api-access-lmtft\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.135707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.135623 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l"] Nov 02 14:22:13.139673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.139658 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.160973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.160946 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:22:13.160973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.160948 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:22:13.164138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.164113 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l"] Nov 02 14:22:13.204513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.204476 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.214285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.214247 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.214431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.214309 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.214489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.214453 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v274\" (UniqueName: \"kubernetes.io/projected/5e9d2f1d-8c50-4e32-be77-e2e92787c597-kube-api-access-8v274\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.315975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.315933 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.316138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.316045 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v274\" (UniqueName: \"kubernetes.io/projected/5e9d2f1d-8c50-4e32-be77-e2e92787c597-kube-api-access-8v274\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.316218 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.316204 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.316968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.316947 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.319850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.319821 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.343187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.343160 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v274\" (UniqueName: \"kubernetes.io/projected/5e9d2f1d-8c50-4e32-be77-e2e92787c597-kube-api-access-8v274\") pod \"maas-api-revoke-sub-43f990fc-sdv8l\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.455955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.454616 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:13.529585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.529557 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr"] Nov 02 14:22:13.579295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.579272 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:22:13.671466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.671436 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" event={"ID":"a947cb83-0905-462f-8b99-dc2da8238f81","Type":"ContainerStarted","Data":"1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b"} Nov 02 14:22:13.671613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.671489 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" event={"ID":"a947cb83-0905-462f-8b99-dc2da8238f81","Type":"ContainerStarted","Data":"f4d743020aff813a06b270c6b9d013f74f936da9c37039ab5b2083ded95c49e8"} Nov 02 14:22:13.672366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.672344 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:13.719901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.719816 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" podStartSLOduration=1.7198012139999999 podStartE2EDuration="1.719801214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:13.716325459 +0000 UTC m=+1498.753430390" watchObservedRunningTime="2026-11-02 14:22:13.719801214 +0000 UTC m=+1498.756906145" Nov 02 14:22:13.777447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:13.777426 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l"] Nov 02 14:22:13.777886 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:13.777861 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9d2f1d_8c50_4e32_be77_e2e92787c597.slice/crio-bf8e02e70799cd9dbbfaa400b9f5b831bd2564d3965cc9faab08841947b1e866 WatchSource:0}: Error finding container bf8e02e70799cd9dbbfaa400b9f5b831bd2564d3965cc9faab08841947b1e866: Status 404 returned error can't find the container with id bf8e02e70799cd9dbbfaa400b9f5b831bd2564d3965cc9faab08841947b1e866 Nov 02 14:22:14.625800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:14.624946 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:22:14.763842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:14.761903 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" event={"ID":"5e9d2f1d-8c50-4e32-be77-e2e92787c597","Type":"ContainerStarted","Data":"ad4c1ee70bd9e61e0d615b52b8eb0932da2fc2a48c9a6166a1ff08e0dd72bab6"} Nov 02 14:22:14.763842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:14.762512 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" event={"ID":"5e9d2f1d-8c50-4e32-be77-e2e92787c597","Type":"ContainerStarted","Data":"bf8e02e70799cd9dbbfaa400b9f5b831bd2564d3965cc9faab08841947b1e866"} Nov 02 14:22:15.834365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:15.834326 4431 generic.go:356] "Generic (PLEG): container finished" podID="5e9d2f1d-8c50-4e32-be77-e2e92787c597" containerID="ad4c1ee70bd9e61e0d615b52b8eb0932da2fc2a48c9a6166a1ff08e0dd72bab6" exitCode=0 Nov 02 14:22:15.836291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:15.835225 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" event={"ID":"5e9d2f1d-8c50-4e32-be77-e2e92787c597","Type":"ContainerDied","Data":"ad4c1ee70bd9e61e0d615b52b8eb0932da2fc2a48c9a6166a1ff08e0dd72bab6"} Nov 02 14:22:16.092867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.092629 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:22:16.092867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.092702 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:22:16.527103 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.527071 4431 scope.go:122] "RemoveContainer" containerID="0e3028ea63cb01339f6e1b44eb35e3a69ef2411def76bae9cfee874407112282" Nov 02 14:22:16.539137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.539117 4431 scope.go:122] "RemoveContainer" containerID="264575987ee30a853503b73c983859159d506b38ea2857d218eaf414a43d0b05" Nov 02 14:22:16.547964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.547940 4431 scope.go:122] "RemoveContainer" containerID="134a2029be470d0465bcd14f7493148058791092c719987985b34028810b207c" Nov 02 14:22:16.556802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.556759 4431 scope.go:122] "RemoveContainer" containerID="23b6589f89bf58bcb2b7331e3ec91d74e79e623e2ebaeb8839e1561fb237fd1a" Nov 02 14:22:16.567758 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.567149 4431 scope.go:122] "RemoveContainer" containerID="faa19b3ef155dcf2a82991dc9ca0907ba034a6e6044baff6e44d5f5266a28c69" Nov 02 14:22:16.576408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.576389 4431 scope.go:122] "RemoveContainer" containerID="9df3b9bb9df87c40fcea8f09b2c255d5667590b1934d238f6abd68a4a1d69bf6" Nov 02 14:22:16.585501 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.585482 4431 scope.go:122] "RemoveContainer" containerID="fb3fc3560139f6024101c00f1f53fb8db4a7f7c8ef52e70b02aaa45fd2b72e75" Nov 02 14:22:16.595504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:16.595472 4431 scope.go:122] "RemoveContainer" containerID="554c0b3df6647cca6e512698fe44fd01f7fd316606b7a2abb3245dc083df2c00" Nov 02 14:22:17.053353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.053328 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:17.171596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.171568 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9d2f1d-8c50-4e32-be77-e2e92787c597-kube-api-access-8v274\" (UniqueName: \"kubernetes.io/projected/5e9d2f1d-8c50-4e32-be77-e2e92787c597-kube-api-access-8v274\") pod \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " Nov 02 14:22:17.171801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.171696 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-openshift-ca\") pod \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " Nov 02 14:22:17.171801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.171753 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-xks-ca\") pod \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\" (UID: \"5e9d2f1d-8c50-4e32-be77-e2e92787c597\") " Nov 02 14:22:17.172139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.172105 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-openshift-ca" pod "5e9d2f1d-8c50-4e32-be77-e2e92787c597" (UID: "5e9d2f1d-8c50-4e32-be77-e2e92787c597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:17.173007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.172980 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:17.174006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.173979 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9d2f1d-8c50-4e32-be77-e2e92787c597-kube-api-access-8v274" pod "5e9d2f1d-8c50-4e32-be77-e2e92787c597" (UID: "5e9d2f1d-8c50-4e32-be77-e2e92787c597"). InnerVolumeSpecName "kube-api-access-8v274". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:17.174481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.174455 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-xks-ca" pod "5e9d2f1d-8c50-4e32-be77-e2e92787c597" (UID: "5e9d2f1d-8c50-4e32-be77-e2e92787c597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:22:17.273849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.273822 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e9d2f1d-8c50-4e32-be77-e2e92787c597-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:17.273849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.273844 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v274\" (UniqueName: \"kubernetes.io/projected/5e9d2f1d-8c50-4e32-be77-e2e92787c597-kube-api-access-8v274\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:17.577145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.577069 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p"] Nov 02 14:22:17.579928 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.579903 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:17.853738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.853667 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" Nov 02 14:22:17.853738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.853693 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l" event={"ID":"5e9d2f1d-8c50-4e32-be77-e2e92787c597","Type":"ContainerDied","Data":"bf8e02e70799cd9dbbfaa400b9f5b831bd2564d3965cc9faab08841947b1e866"} Nov 02 14:22:17.854006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:17.853744 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf8e02e70799cd9dbbfaa400b9f5b831bd2564d3965cc9faab08841947b1e866" Nov 02 14:22:21.296466 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:21.296434 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:22:22.578172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.578112 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" podUID="1d97b73d-9190-44cb-b11a-be2421ca0759" containerName="payload-processing" containerID="cri-o://b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9" gracePeriod=25 Nov 02 14:22:22.847300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.847242 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:22.890709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.890660 4431 generic.go:356] "Generic (PLEG): container finished" podID="1d97b73d-9190-44cb-b11a-be2421ca0759" containerID="b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9" exitCode=0 Nov 02 14:22:22.890709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.890705 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" Nov 02 14:22:22.890905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.890742 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" event={"ID":"1d97b73d-9190-44cb-b11a-be2421ca0759","Type":"ContainerDied","Data":"b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9"} Nov 02 14:22:22.890905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.890826 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p" event={"ID":"1d97b73d-9190-44cb-b11a-be2421ca0759","Type":"ContainerDied","Data":"daeced38d2b6dcb915c4d47ca5866646fc812afaebc558002f15eae1b0c79796"} Nov 02 14:22:22.890905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.890852 4431 scope.go:122] "RemoveContainer" containerID="b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9" Nov 02 14:22:22.902487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.902464 4431 scope.go:122] "RemoveContainer" containerID="b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9" Nov 02 14:22:22.902831 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:22.902805 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9\": container with ID starting with b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9 not found: ID does not exist" containerID="b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9" Nov 02 14:22:22.903022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.902838 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9"} err="failed to get container status \"b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9\": rpc error: code = NotFound desc = could not find container \"b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9\": container with ID starting with b4202d3ed0f76519eb426c29fe013fe943ab262d61b156abd1cecaa89e2608b9 not found: ID does not exist" Nov 02 14:22:22.949059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.949033 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-routing-overlay-volume\") pod \"1d97b73d-9190-44cb-b11a-be2421ca0759\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " Nov 02 14:22:22.949165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.949141 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-plugins-config-volume\") pod \"1d97b73d-9190-44cb-b11a-be2421ca0759\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " Nov 02 14:22:22.949206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.949189 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-kube-api-access-kssx2\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-kube-api-access-kssx2\") pod \"1d97b73d-9190-44cb-b11a-be2421ca0759\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " Nov 02 14:22:22.949268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.949248 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-provider-credentials\") pod \"1d97b73d-9190-44cb-b11a-be2421ca0759\" (UID: \"1d97b73d-9190-44cb-b11a-be2421ca0759\") " Nov 02 14:22:22.949426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.949402 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-routing-overlay-volume" pod "1d97b73d-9190-44cb-b11a-be2421ca0759" (UID: "1d97b73d-9190-44cb-b11a-be2421ca0759"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:22.949529 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.949510 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-plugins-config-volume" pod "1d97b73d-9190-44cb-b11a-be2421ca0759" (UID: "1d97b73d-9190-44cb-b11a-be2421ca0759"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:22.949683 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.949668 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:22.949724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.949690 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d97b73d-9190-44cb-b11a-be2421ca0759-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:22.951343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.951323 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-kube-api-access-kssx2" pod "1d97b73d-9190-44cb-b11a-be2421ca0759" (UID: "1d97b73d-9190-44cb-b11a-be2421ca0759"). InnerVolumeSpecName "kube-api-access-kssx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:22.951438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:22.951421 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-provider-credentials" pod "1d97b73d-9190-44cb-b11a-be2421ca0759" (UID: "1d97b73d-9190-44cb-b11a-be2421ca0759"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:23.051289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.051261 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kssx2\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-kube-api-access-kssx2\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:23.051289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.051287 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d97b73d-9190-44cb-b11a-be2421ca0759-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:23.236981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.236947 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p"] Nov 02 14:22:23.240244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.240217 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-86w9p"] Nov 02 14:22:23.461838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.461797 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5"] Nov 02 14:22:23.462350 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.462337 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1d97b73d-9190-44cb-b11a-be2421ca0759" containerName="payload-processing" Nov 02 14:22:23.462402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.462351 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d97b73d-9190-44cb-b11a-be2421ca0759" containerName="payload-processing" Nov 02 14:22:23.462402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.462376 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5e9d2f1d-8c50-4e32-be77-e2e92787c597" containerName="revoke-keys" Nov 02 14:22:23.462402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.462381 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9d2f1d-8c50-4e32-be77-e2e92787c597" containerName="revoke-keys" Nov 02 14:22:23.462494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.462470 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9d2f1d-8c50-4e32-be77-e2e92787c597" containerName="revoke-keys" Nov 02 14:22:23.462494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.462485 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d97b73d-9190-44cb-b11a-be2421ca0759" containerName="payload-processing" Nov 02 14:22:23.467159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.467138 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.483446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.483418 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 14:22:23.483578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.483540 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 14:22:23.483642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.483606 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 14:22:23.483698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.483669 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 14:22:23.486456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.486424 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5"] Nov 02 14:22:23.495548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.495484 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1a9273a49ced\"" Nov 02 14:22:23.495548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.495490 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gbmzv\"" Nov 02 14:22:23.557496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.557469 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-plugins-config-volume\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.557665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.557643 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-routing-overlay-volume\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.557722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.557689 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-provider-credentials\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.557799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.557768 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g55n6\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-kube-api-access-g55n6\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.659004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.658972 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g55n6\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-kube-api-access-g55n6\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.659399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.659050 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-plugins-config-volume\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.659399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.659188 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-routing-overlay-volume\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.659399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.659234 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-provider-credentials\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.659955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.659929 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-routing-overlay-volume\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.659955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.659948 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-plugins-config-volume\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.661623 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.661604 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-provider-credentials\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.684588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.684558 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g55n6\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-kube-api-access-g55n6\") pod \"payload-processing-external-model-85d6f9ccbb-pssz5\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:23.781659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:23.781569 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:24.093224 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:24.093199 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5"] Nov 02 14:22:24.093990 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:24.093968 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134be206_8c24_4dbf_82e8_7a2874836236.slice/crio-34443fb7fe20a767f552c1581229170d0bfbd75e40fb5d70986d735e53d9ea68 WatchSource:0}: Error finding container 34443fb7fe20a767f552c1581229170d0bfbd75e40fb5d70986d735e53d9ea68: Status 404 returned error can't find the container with id 34443fb7fe20a767f552c1581229170d0bfbd75e40fb5d70986d735e53d9ea68 Nov 02 14:22:24.911507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:24.911473 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" event={"ID":"134be206-8c24-4dbf-82e8-7a2874836236","Type":"ContainerStarted","Data":"7f3ccdecc58a18c26fa134e027391ba10c822044d2a351a3e287bbdd5f60b940"} Nov 02 14:22:24.911958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:24.911526 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" event={"ID":"134be206-8c24-4dbf-82e8-7a2874836236","Type":"ContainerStarted","Data":"34443fb7fe20a767f552c1581229170d0bfbd75e40fb5d70986d735e53d9ea68"} Nov 02 14:22:24.911958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:24.911565 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:24.958638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:24.958590 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" podStartSLOduration=1.9585786889999999 podStartE2EDuration="1.958578689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:24.955987342 +0000 UTC m=+1509.993092275" watchObservedRunningTime="2026-11-02 14:22:24.958578689 +0000 UTC m=+1509.995683662" Nov 02 14:22:25.078880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:25.078633 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d97b73d-9190-44cb-b11a-be2421ca0759" path="/var/lib/kubelet/pods/1d97b73d-9190-44cb-b11a-be2421ca0759/volumes" Nov 02 14:22:25.838960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:25.838935 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:22:27.006466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.006432 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44"] Nov 02 14:22:27.708731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.708691 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5"] Nov 02 14:22:27.737799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.737227 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8"] Nov 02 14:22:27.772301 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.772262 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8"] Nov 02 14:22:27.772485 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.772393 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.792045 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.792016 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-623b4b-openshift-default-dockercfg-bw4ld\"" Nov 02 14:22:27.915918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.915883 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-certs\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.916069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.916041 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.916148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.916079 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-istio-token\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.916148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.916115 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-envoy\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.916252 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.916233 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-credential-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.916288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.916268 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-data\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.916336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.916293 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a947c82c-52dc-487d-a500-d6402419accb-istio-podinfo\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.916393 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.916330 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a947c82c-52dc-487d-a500-d6402419accb-istiod-ca-cert\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:27.916459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:27.916428 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxv6f\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-kube-api-access-sxv6f\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017059 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-certs\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017180 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017208 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-istio-token\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017234 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-envoy\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017267 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-credential-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017310 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-data\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017348 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a947c82c-52dc-487d-a500-d6402419accb-istio-podinfo\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017457 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a947c82c-52dc-487d-a500-d6402419accb-istiod-ca-cert\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017503 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-certs\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.017958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.017544 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxv6f\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-kube-api-access-sxv6f\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.018158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.018136 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-data\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.018251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.018207 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-credential-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.018299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.018258 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-socket\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.018299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.018259 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a947c82c-52dc-487d-a500-d6402419accb-istiod-ca-cert\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.019641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.019619 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-envoy\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.019890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.019868 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a947c82c-52dc-487d-a500-d6402419accb-istio-podinfo\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.041942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.041917 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-istio-token\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.042112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.042094 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxv6f\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-kube-api-access-sxv6f\") pod \"e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.116076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.115799 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:28.516794 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:28.516746 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda947c82c_52dc_487d_a500_d6402419accb.slice/crio-ec42bfa5279943cf1060f6ec7b0d577cfa158c7651fe64b595f55b2bcea98e0d WatchSource:0}: Error finding container ec42bfa5279943cf1060f6ec7b0d577cfa158c7651fe64b595f55b2bcea98e0d: Status 404 returned error can't find the container with id ec42bfa5279943cf1060f6ec7b0d577cfa158c7651fe64b595f55b2bcea98e0d Nov 02 14:22:28.516926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.516884 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8"] Nov 02 14:22:28.519548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.519516 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:28.519638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.519578 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:28.519638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.519606 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:28.944267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.944226 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" event={"ID":"a947c82c-52dc-487d-a500-d6402419accb","Type":"ContainerStarted","Data":"8c7de268a28655066b1081b212027956cfebeec13ef1734be85b9c7a172fd844"} Nov 02 14:22:28.944446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.944326 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" event={"ID":"a947c82c-52dc-487d-a500-d6402419accb","Type":"ContainerStarted","Data":"ec42bfa5279943cf1060f6ec7b0d577cfa158c7651fe64b595f55b2bcea98e0d"} Nov 02 14:22:28.992670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:28.992613 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podStartSLOduration=1.992597238 podStartE2EDuration="1.992597238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:28.991954237 +0000 UTC m=+1514.029059181" watchObservedRunningTime="2026-11-02 14:22:28.992597238 +0000 UTC m=+1514.029702197" Nov 02 14:22:29.117208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:29.117178 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:29.118735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:29.118711 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 14:22:29.118858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:29.118751 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 14:22:30.116901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:30.116867 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 14:22:30.117118 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:30.116922 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 14:22:31.124480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:31.124447 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 14:22:31.125080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:31.125050 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 14:22:32.117267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:32.117231 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 14:22:32.117467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:32.117411 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 14:22:32.710008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:32.709962 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" podUID="134be206-8c24-4dbf-82e8-7a2874836236" containerName="payload-processing" containerID="cri-o://7f3ccdecc58a18c26fa134e027391ba10c822044d2a351a3e287bbdd5f60b940" gracePeriod=25 Nov 02 14:22:32.840962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:32.840934 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:22:33.051808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.051716 4431 generic.go:356] "Generic (PLEG): container finished" podID="134be206-8c24-4dbf-82e8-7a2874836236" containerID="7f3ccdecc58a18c26fa134e027391ba10c822044d2a351a3e287bbdd5f60b940" exitCode=0 Nov 02 14:22:33.051959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.051834 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" event={"ID":"134be206-8c24-4dbf-82e8-7a2874836236","Type":"ContainerDied","Data":"7f3ccdecc58a18c26fa134e027391ba10c822044d2a351a3e287bbdd5f60b940"} Nov 02 14:22:33.118950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.118908 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 14:22:33.119123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.118995 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 14:22:33.156400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.156378 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:33.330808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.330715 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-plugins-config-volume\") pod \"134be206-8c24-4dbf-82e8-7a2874836236\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " Nov 02 14:22:33.330808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.330769 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-provider-credentials\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-provider-credentials\") pod \"134be206-8c24-4dbf-82e8-7a2874836236\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " Nov 02 14:22:33.331072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.330858 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-kube-api-access-g55n6\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-kube-api-access-g55n6\") pod \"134be206-8c24-4dbf-82e8-7a2874836236\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " Nov 02 14:22:33.331072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.330983 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-routing-overlay-volume\") pod \"134be206-8c24-4dbf-82e8-7a2874836236\" (UID: \"134be206-8c24-4dbf-82e8-7a2874836236\") " Nov 02 14:22:33.331208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.331180 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-plugins-config-volume" pod "134be206-8c24-4dbf-82e8-7a2874836236" (UID: "134be206-8c24-4dbf-82e8-7a2874836236"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:33.331315 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.331299 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:33.331521 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.331497 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-routing-overlay-volume" pod "134be206-8c24-4dbf-82e8-7a2874836236" (UID: "134be206-8c24-4dbf-82e8-7a2874836236"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:33.334147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.333900 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-provider-credentials" pod "134be206-8c24-4dbf-82e8-7a2874836236" (UID: "134be206-8c24-4dbf-82e8-7a2874836236"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:33.334147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.333921 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-kube-api-access-g55n6" pod "134be206-8c24-4dbf-82e8-7a2874836236" (UID: "134be206-8c24-4dbf-82e8-7a2874836236"). InnerVolumeSpecName "kube-api-access-g55n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:33.432432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.432395 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/134be206-8c24-4dbf-82e8-7a2874836236-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:33.432432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.432427 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:33.432642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:33.432441 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g55n6\" (UniqueName: \"kubernetes.io/projected/134be206-8c24-4dbf-82e8-7a2874836236-kube-api-access-g55n6\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:34.088305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:34.088268 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" event={"ID":"134be206-8c24-4dbf-82e8-7a2874836236","Type":"ContainerDied","Data":"34443fb7fe20a767f552c1581229170d0bfbd75e40fb5d70986d735e53d9ea68"} Nov 02 14:22:34.088918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:34.088896 4431 scope.go:122] "RemoveContainer" containerID="7f3ccdecc58a18c26fa134e027391ba10c822044d2a351a3e287bbdd5f60b940" Nov 02 14:22:34.089220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:34.089201 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5" Nov 02 14:22:34.117049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:34.117016 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 14:22:34.117257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:34.117232 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 14:22:34.282803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:34.269886 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5"] Nov 02 14:22:34.321968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:34.321933 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-85d6f9ccbb-pssz5"] Nov 02 14:22:35.083807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:35.082647 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134be206-8c24-4dbf-82e8-7a2874836236" path="/var/lib/kubelet/pods/134be206-8c24-4dbf-82e8-7a2874836236/volumes" Nov 02 14:22:35.141801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:35.141143 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 02 14:22:35.141801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:35.141198 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 02 14:22:36.298802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:36.297821 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:22:36.298802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:36.298674 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:22:36.318800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:36.316927 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:22:36.362800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:36.360887 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:22:36.382800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:36.376806 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Nov 02 14:22:36.497801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:36.496868 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:22:37.163802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.163089 4431 patch_prober.go:29] interesting pod/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:22:37.163802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.163170 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:22:37.184801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.175855 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:22:37.378800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.362409 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:22:37.382192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.382166 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:22:37.418323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.418243 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:22:37.774445 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.772371 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:37.794876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.794852 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:37.796408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:37.796388 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:22:38.261804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:38.260690 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx"] Nov 02 14:22:38.289803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:38.286495 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4082dcca-d7hmx"] Nov 02 14:22:39.101172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:39.101142 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14b5da6-876d-4af9-a5b3-7cb995708cca" path="/var/lib/kubelet/pods/e14b5da6-876d-4af9-a5b3-7cb995708cca/volumes" Nov 02 14:22:42.032956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.032919 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" containerID="cri-o://a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2" gracePeriod=45 Nov 02 14:22:42.333401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.333372 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:22:42.439911 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.439872 4431 generic.go:356] "Generic (PLEG): container finished" podID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerID="a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2" exitCode=0 Nov 02 14:22:42.440114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.440097 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" Nov 02 14:22:42.440182 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.440126 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" event={"ID":"04f711d0-98ec-4d7f-936b-b3d223cba1b2","Type":"ContainerDied","Data":"a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2"} Nov 02 14:22:42.440227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.440196 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" event={"ID":"04f711d0-98ec-4d7f-936b-b3d223cba1b2","Type":"ContainerDied","Data":"f310e642c2abe15ada6399818e8673caae5fe0398bbd2431fcda3df317ba0603"} Nov 02 14:22:42.440227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.440219 4431 scope.go:122] "RemoveContainer" containerID="a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2" Nov 02 14:22:42.442666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.442638 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tls-certs\") pod \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " Nov 02 14:22:42.442845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.442733 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f711d0-98ec-4d7f-936b-b3d223cba1b2-kube-api-access-rxdl9\" (UniqueName: \"kubernetes.io/projected/04f711d0-98ec-4d7f-936b-b3d223cba1b2-kube-api-access-rxdl9\") pod \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " Nov 02 14:22:42.442845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.442819 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-dshm\") pod \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " Nov 02 14:22:42.442981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.442955 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-home\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-home\") pod \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " Nov 02 14:22:42.443045 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.442984 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tmp-dir\") pod \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " Nov 02 14:22:42.443045 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.443019 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-model-cache\") pod \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\" (UID: \"04f711d0-98ec-4d7f-936b-b3d223cba1b2\") " Nov 02 14:22:42.443754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.443448 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tmp-dir" pod "04f711d0-98ec-4d7f-936b-b3d223cba1b2" (UID: "04f711d0-98ec-4d7f-936b-b3d223cba1b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:42.443754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.443608 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:42.443754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.443704 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-home" pod "04f711d0-98ec-4d7f-936b-b3d223cba1b2" (UID: "04f711d0-98ec-4d7f-936b-b3d223cba1b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:42.443754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.443716 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-model-cache" pod "04f711d0-98ec-4d7f-936b-b3d223cba1b2" (UID: "04f711d0-98ec-4d7f-936b-b3d223cba1b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:42.445247 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.445224 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tls-certs" pod "04f711d0-98ec-4d7f-936b-b3d223cba1b2" (UID: "04f711d0-98ec-4d7f-936b-b3d223cba1b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:22:42.445925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.445901 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f711d0-98ec-4d7f-936b-b3d223cba1b2-kube-api-access-rxdl9" pod "04f711d0-98ec-4d7f-936b-b3d223cba1b2" (UID: "04f711d0-98ec-4d7f-936b-b3d223cba1b2"). InnerVolumeSpecName "kube-api-access-rxdl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:42.446336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.446313 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-dshm" pod "04f711d0-98ec-4d7f-936b-b3d223cba1b2" (UID: "04f711d0-98ec-4d7f-936b-b3d223cba1b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:42.451503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.451485 4431 scope.go:122] "RemoveContainer" containerID="a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2" Nov 02 14:22:42.451820 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:42.451798 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2\": container with ID starting with a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2 not found: ID does not exist" containerID="a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2" Nov 02 14:22:42.451890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.451828 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2"} err="failed to get container status \"a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2\": rpc error: code = NotFound desc = could not find container \"a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2\": container with ID starting with a6a41153b6eff87de7c2d10f75282555dcb41bc85f9486ce6404e1ab5ef358c2 not found: ID does not exist" Nov 02 14:22:42.544879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.544846 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f711d0-98ec-4d7f-936b-b3d223cba1b2-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:42.544879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.544874 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdl9\" (UniqueName: \"kubernetes.io/projected/04f711d0-98ec-4d7f-936b-b3d223cba1b2-kube-api-access-rxdl9\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:42.544879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.544884 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:42.545111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.544893 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:42.545111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.544901 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f711d0-98ec-4d7f-936b-b3d223cba1b2-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:42.794901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.794870 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44"] Nov 02 14:22:42.806216 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:42.806184 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44"] Nov 02 14:22:43.076351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:43.076281 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" path="/var/lib/kubelet/pods/04f711d0-98ec-4d7f-936b-b3d223cba1b2/volumes" Nov 02 14:22:43.272343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:43.272303 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/auth-test-model-f1e1c2-kserve-6b4f97f56f-56q44" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.130:8000/ready\": context deadline exceeded" Nov 02 14:22:44.208842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:44.208795 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v"] Nov 02 14:22:44.836931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:44.836289 4431 patch_prober.go:29] interesting pod/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": read tcp 10.133.0.2:33350->10.133.0.143:15021: read: connection reset by peer" start-of-body= Nov 02 14:22:44.836931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:44.836354 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": read tcp 10.133.0.2:33350->10.133.0.143:15021: read: connection reset by peer" Nov 02 14:22:45.464735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:45.464703 4431 generic.go:356] "Generic (PLEG): container finished" podID="62790564-506b-4f53-a9ab-582b2565bc29" containerID="9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5" exitCode=0 Nov 02 14:22:45.465160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:45.464786 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" event={"ID":"62790564-506b-4f53-a9ab-582b2565bc29","Type":"ContainerDied","Data":"9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5"} Nov 02 14:22:45.465160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:45.465145 4431 scope.go:122] "RemoveContainer" containerID="9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5" Nov 02 14:22:45.466650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:45.466623 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:45.466716 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:45.466689 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:45.466752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:45.466726 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:46.497920 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:46.497883 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" event={"ID":"62790564-506b-4f53-a9ab-582b2565bc29","Type":"ContainerStarted","Data":"06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2"} Nov 02 14:22:46.784089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:46.782176 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:22:46.784089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:46.782308 4431 patch_prober.go:29] interesting pod/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:22:46.784089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:46.782351 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:22:47.751819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.749096 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x"] Nov 02 14:22:47.751819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.749985 4431 cpu_manager.go:414] "containerMap: removing container" podUID="134be206-8c24-4dbf-82e8-7a2874836236" containerName="payload-processing" Nov 02 14:22:47.751819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.750004 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134be206-8c24-4dbf-82e8-7a2874836236" containerName="payload-processing" Nov 02 14:22:47.751819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.750064 4431 cpu_manager.go:414] "containerMap: removing container" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" Nov 02 14:22:47.751819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.750072 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" Nov 02 14:22:47.751819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.750223 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f711d0-98ec-4d7f-936b-b3d223cba1b2" containerName="main" Nov 02 14:22:47.751819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.750243 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="134be206-8c24-4dbf-82e8-7a2874836236" containerName="payload-processing" Nov 02 14:22:47.753671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.753648 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.773631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.773608 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x"] Nov 02 14:22:47.791798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.783056 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-944506-openshift-default-dockercfg-wz7jz\"" Nov 02 14:22:47.791798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.783999 4431 patch_prober.go:29] interesting pod/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:22:47.791798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.784043 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:22:47.818073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818042 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-podinfo\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.818261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818085 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-token\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.818261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818151 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-credential-socket\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.818261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818185 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-certs\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.818261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818224 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-data\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.818433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818265 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c49wf\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-kube-api-access-c49wf\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.818433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818337 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istiod-ca-cert\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.818433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818395 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-socket\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.818535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.818465 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-envoy\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.923799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.923095 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-credential-socket\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.923799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.923641 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-certs\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.923799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.923685 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-data\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.923799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.923726 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c49wf\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-kube-api-access-c49wf\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.924339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.923582 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-credential-socket\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.924462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.924356 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-certs\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.924616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.924593 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-data\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.934799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.925460 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istiod-ca-cert\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.934956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.924604 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istiod-ca-cert\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.954801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.936063 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-socket\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.955127 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.955099 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-envoy\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.955397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.955376 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-podinfo\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.955539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.955521 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-token\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.956220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.938934 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-socket\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.965283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.965123 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-envoy\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:47.977794 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.965664 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-podinfo\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:48.002852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:47.994728 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c49wf\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-kube-api-access-c49wf\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:48.051364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.051252 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-token\") pod \"e2e-extmod-a-944506-openshift-default-855597c485-fdk9x\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:48.097453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.075057 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:48.435900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.435865 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x"] Nov 02 14:22:48.436410 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:48.436379 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode486b7e9_6294_4ff0_b79e_522f2fbdc4f0.slice/crio-2e03c5013f8d6bc1049d5af25ea272056ab611c5e28e77817879ae14b81f2686 WatchSource:0}: Error finding container 2e03c5013f8d6bc1049d5af25ea272056ab611c5e28e77817879ae14b81f2686: Status 404 returned error can't find the container with id 2e03c5013f8d6bc1049d5af25ea272056ab611c5e28e77817879ae14b81f2686 Nov 02 14:22:48.439387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.439350 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:48.439466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.439437 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:48.439507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.439481 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:48.521817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.520264 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" event={"ID":"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0","Type":"ContainerStarted","Data":"2e03c5013f8d6bc1049d5af25ea272056ab611c5e28e77817879ae14b81f2686"} Nov 02 14:22:48.775847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.775562 4431 patch_prober.go:29] interesting pod/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:22:48.775847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:48.775618 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:22:49.528866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:49.528818 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" event={"ID":"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0","Type":"ContainerStarted","Data":"da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba"} Nov 02 14:22:49.656422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:49.656277 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podStartSLOduration=2.656260395 podStartE2EDuration="2.656260395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:49.641026898 +0000 UTC m=+1534.678131842" watchObservedRunningTime="2026-11-02 14:22:49.656260395 +0000 UTC m=+1534.693365328" Nov 02 14:22:49.780851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:49.779193 4431 patch_prober.go:29] interesting pod/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:22:49.780851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:49.779258 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:22:50.091856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.086021 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:50.098943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.098916 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:50.536131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.536103 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:50.537150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.537127 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:22:50.650107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.650077 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c"] Nov 02 14:22:50.654288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.654268 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.707846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.698125 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:22:50.708354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.708308 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:22:50.709964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.709938 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c"] Nov 02 14:22:50.718060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.718033 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.718169 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.718109 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8svr\" (UniqueName: \"kubernetes.io/projected/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-kube-api-access-q8svr\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.718299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.718286 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.797174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.797101 4431 patch_prober.go:29] interesting pod/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 02 14:22:50.797174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.797159 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 02 14:22:50.819505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.819479 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.819637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.819532 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8svr\" (UniqueName: \"kubernetes.io/projected/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-kube-api-access-q8svr\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.819741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.819724 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.820413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.820392 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.821883 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.821865 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.844827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.844805 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8svr\" (UniqueName: \"kubernetes.io/projected/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-kube-api-access-q8svr\") pod \"maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:50.968962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:50.968932 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:51.036654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:51.036601 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv"] Nov 02 14:22:51.041306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:51.041278 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-333805c3-wb4mv"] Nov 02 14:22:51.108934 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:51.108896 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58577ac1-c47c-4730-8c15-688c84f0e39a" path="/var/lib/kubelet/pods/58577ac1-c47c-4730-8c15-688c84f0e39a/volumes" Nov 02 14:22:51.307435 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:51.307404 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:22:51.309209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:51.309185 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c"] Nov 02 14:22:51.309940 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:51.309912 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a64636c_8c1b_4f32_aca7_b508c4d17fcf.slice/crio-2e89f3020cd3ebc69829a5aa91e2360aef706673b2899e9762204d2fd0fc2b48 WatchSource:0}: Error finding container 2e89f3020cd3ebc69829a5aa91e2360aef706673b2899e9762204d2fd0fc2b48: Status 404 returned error can't find the container with id 2e89f3020cd3ebc69829a5aa91e2360aef706673b2899e9762204d2fd0fc2b48 Nov 02 14:22:51.545220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:51.545182 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" event={"ID":"4a64636c-8c1b-4f32-aca7-b508c4d17fcf","Type":"ContainerStarted","Data":"2e89f3020cd3ebc69829a5aa91e2360aef706673b2899e9762204d2fd0fc2b48"} Nov 02 14:22:51.779957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:51.779935 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:22:51.780369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:51.780347 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:22:51.781000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:51.780982 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:22:52.554608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:52.554522 4431 generic.go:356] "Generic (PLEG): container finished" podID="4a64636c-8c1b-4f32-aca7-b508c4d17fcf" containerID="c5c0a59ffb988f42f30b3ad4aa5b53617ec95805dc4333e5a827123caba3abcc" exitCode=0 Nov 02 14:22:52.554608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:52.554588 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" event={"ID":"4a64636c-8c1b-4f32-aca7-b508c4d17fcf","Type":"ContainerDied","Data":"c5c0a59ffb988f42f30b3ad4aa5b53617ec95805dc4333e5a827123caba3abcc"} Nov 02 14:22:53.723000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.722972 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:53.860524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.860442 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-kube-api-access-q8svr\" (UniqueName: \"kubernetes.io/projected/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-kube-api-access-q8svr\") pod \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " Nov 02 14:22:53.860524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.860513 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-openshift-ca\") pod \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " Nov 02 14:22:53.860754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.860579 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-xks-ca\") pod \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\" (UID: \"4a64636c-8c1b-4f32-aca7-b508c4d17fcf\") " Nov 02 14:22:53.860903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.860880 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-openshift-ca" pod "4a64636c-8c1b-4f32-aca7-b508c4d17fcf" (UID: "4a64636c-8c1b-4f32-aca7-b508c4d17fcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:22:53.861064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.861051 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:53.862610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.862575 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-xks-ca" pod "4a64636c-8c1b-4f32-aca7-b508c4d17fcf" (UID: "4a64636c-8c1b-4f32-aca7-b508c4d17fcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:22:53.862830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.862808 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-kube-api-access-q8svr" pod "4a64636c-8c1b-4f32-aca7-b508c4d17fcf" (UID: "4a64636c-8c1b-4f32-aca7-b508c4d17fcf"). InnerVolumeSpecName "kube-api-access-q8svr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:53.961937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.961907 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8svr\" (UniqueName: \"kubernetes.io/projected/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-kube-api-access-q8svr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:53.961937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:53.961931 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a64636c-8c1b-4f32-aca7-b508c4d17fcf-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:54.569226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:54.569197 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" event={"ID":"4a64636c-8c1b-4f32-aca7-b508c4d17fcf","Type":"ContainerDied","Data":"2e89f3020cd3ebc69829a5aa91e2360aef706673b2899e9762204d2fd0fc2b48"} Nov 02 14:22:54.569226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:54.569221 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c" Nov 02 14:22:54.569431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:54.569242 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e89f3020cd3ebc69829a5aa91e2360aef706673b2899e9762204d2fd0fc2b48" Nov 02 14:22:56.910445 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:56.910411 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526"] Nov 02 14:22:56.910969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:56.910953 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4a64636c-8c1b-4f32-aca7-b508c4d17fcf" containerName="revoke-keys" Nov 02 14:22:56.911083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:56.910969 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a64636c-8c1b-4f32-aca7-b508c4d17fcf" containerName="revoke-keys" Nov 02 14:22:56.911146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:56.911103 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a64636c-8c1b-4f32-aca7-b508c4d17fcf" containerName="revoke-keys" Nov 02 14:22:56.914634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:56.914616 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:56.932162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:56.932145 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-12b8da-openshift-default-dockercfg-qr7jf\"" Nov 02 14:22:56.949922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:56.949897 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526"] Nov 02 14:22:57.108441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.108402 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istiod-ca-cert\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.108441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.108449 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-podinfo\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.108729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.108591 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-certs\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.108825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.108735 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqn7f\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-kube-api-access-hqn7f\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.108937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.108833 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-envoy\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.108937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.108870 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-data\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.108937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.108908 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-credential-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.109133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.108989 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.109133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.109018 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-token\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.210609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.210529 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.210609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.210564 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-token\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.210902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.210706 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istiod-ca-cert\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.210902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.210760 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-podinfo\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.210902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.210880 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-certs\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.210968 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqn7f\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-kube-api-access-hqn7f\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.210976 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.211051 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-envoy\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.211089 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-data\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.211136 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-credential-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.211180 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-certs\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.211423 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istiod-ca-cert\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.211455 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-data\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.211555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.211482 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-credential-socket\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.213318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.213292 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-podinfo\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.213405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.213321 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-envoy\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.238335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.238306 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-token\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.240003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.239984 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqn7f\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-kube-api-access-hqn7f\") pod \"e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.528615 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.528550 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:57.851225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.851196 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526"] Nov 02 14:22:57.851833 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:22:57.851805 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cfed86_5e58_4292_8f84_4ee7ff3b5b76.slice/crio-94eccbe73ad626eff036d4f73494fc4fe33410031f7ee5ba760622205b701359 WatchSource:0}: Error finding container 94eccbe73ad626eff036d4f73494fc4fe33410031f7ee5ba760622205b701359: Status 404 returned error can't find the container with id 94eccbe73ad626eff036d4f73494fc4fe33410031f7ee5ba760622205b701359 Nov 02 14:22:57.854345 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.854310 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:57.854427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.854376 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:57.854427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:57.854407 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:22:58.624804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:58.623357 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" event={"ID":"43cfed86-5e58-4292-8f84-4ee7ff3b5b76","Type":"ContainerStarted","Data":"0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748"} Nov 02 14:22:58.624804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:58.623424 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" event={"ID":"43cfed86-5e58-4292-8f84-4ee7ff3b5b76","Type":"ContainerStarted","Data":"94eccbe73ad626eff036d4f73494fc4fe33410031f7ee5ba760622205b701359"} Nov 02 14:22:58.728805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:58.725646 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" podStartSLOduration=2.725593539 podStartE2EDuration="2.725593539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:58.687924737 +0000 UTC m=+1543.725029670" watchObservedRunningTime="2026-11-02 14:22:58.725593539 +0000 UTC m=+1543.762698474" Nov 02 14:22:59.234608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.234564 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerName="main" containerID="cri-o://ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7" gracePeriod=45 Nov 02 14:22:59.313723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.313684 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": dial tcp 10.133.0.135:8000: connect: connection refused" Nov 02 14:22:59.529431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.529354 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:22:59.530857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.530833 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:22:59.530969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.530887 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:22:59.558747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.558726 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:22:59.633674 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.633640 4431 generic.go:356] "Generic (PLEG): container finished" podID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerID="ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7" exitCode=0 Nov 02 14:22:59.634185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.633679 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" Nov 02 14:22:59.634185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.633727 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" event={"ID":"3079ecc4-301f-482a-ae74-5234a2a97c6c","Type":"ContainerDied","Data":"ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7"} Nov 02 14:22:59.634185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.633806 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v" event={"ID":"3079ecc4-301f-482a-ae74-5234a2a97c6c","Type":"ContainerDied","Data":"322350617cd3021e3bc9f87bd6a61c76107c7a1a3cf3853f470322ba046de458"} Nov 02 14:22:59.634185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.633835 4431 scope.go:122] "RemoveContainer" containerID="ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7" Nov 02 14:22:59.647140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.647122 4431 scope.go:122] "RemoveContainer" containerID="ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7" Nov 02 14:22:59.647424 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:22:59.647404 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7\": container with ID starting with ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7 not found: ID does not exist" containerID="ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7" Nov 02 14:22:59.647479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.647433 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7"} err="failed to get container status \"ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7\": rpc error: code = NotFound desc = could not find container \"ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7\": container with ID starting with ff8ccfc2b12a5948f5ba9d23176dd9fb03017ae69a0e0fc0be3bfb2f8e8fa3b7 not found: ID does not exist" Nov 02 14:22:59.656196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656173 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-model-cache\") pod \"3079ecc4-301f-482a-ae74-5234a2a97c6c\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " Nov 02 14:22:59.656316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656245 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3079ecc4-301f-482a-ae74-5234a2a97c6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3079ecc4-301f-482a-ae74-5234a2a97c6c-tls-certs\") pod \"3079ecc4-301f-482a-ae74-5234a2a97c6c\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " Nov 02 14:22:59.656385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656346 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-dshm\") pod \"3079ecc4-301f-482a-ae74-5234a2a97c6c\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " Nov 02 14:22:59.656447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656435 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-model-cache" pod "3079ecc4-301f-482a-ae74-5234a2a97c6c" (UID: "3079ecc4-301f-482a-ae74-5234a2a97c6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:59.656505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656457 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-home\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-home\") pod \"3079ecc4-301f-482a-ae74-5234a2a97c6c\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " Nov 02 14:22:59.656564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656504 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3079ecc4-301f-482a-ae74-5234a2a97c6c-kube-api-access-lj6q6\" (UniqueName: \"kubernetes.io/projected/3079ecc4-301f-482a-ae74-5234a2a97c6c-kube-api-access-lj6q6\") pod \"3079ecc4-301f-482a-ae74-5234a2a97c6c\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " Nov 02 14:22:59.656564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656540 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-tmp-dir\") pod \"3079ecc4-301f-482a-ae74-5234a2a97c6c\" (UID: \"3079ecc4-301f-482a-ae74-5234a2a97c6c\") " Nov 02 14:22:59.656832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656765 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-home" pod "3079ecc4-301f-482a-ae74-5234a2a97c6c" (UID: "3079ecc4-301f-482a-ae74-5234a2a97c6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:59.656960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.656936 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-tmp-dir" pod "3079ecc4-301f-482a-ae74-5234a2a97c6c" (UID: "3079ecc4-301f-482a-ae74-5234a2a97c6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:59.657185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.657163 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:59.657275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.657190 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:59.657275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.657203 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:59.659405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.659382 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-dshm" pod "3079ecc4-301f-482a-ae74-5234a2a97c6c" (UID: "3079ecc4-301f-482a-ae74-5234a2a97c6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:22:59.659733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.659712 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3079ecc4-301f-482a-ae74-5234a2a97c6c-kube-api-access-lj6q6" pod "3079ecc4-301f-482a-ae74-5234a2a97c6c" (UID: "3079ecc4-301f-482a-ae74-5234a2a97c6c"). InnerVolumeSpecName "kube-api-access-lj6q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:22:59.659991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.659841 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3079ecc4-301f-482a-ae74-5234a2a97c6c-tls-certs" pod "3079ecc4-301f-482a-ae74-5234a2a97c6c" (UID: "3079ecc4-301f-482a-ae74-5234a2a97c6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:22:59.758131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.758104 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj6q6\" (UniqueName: \"kubernetes.io/projected/3079ecc4-301f-482a-ae74-5234a2a97c6c-kube-api-access-lj6q6\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:59.758131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.758128 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3079ecc4-301f-482a-ae74-5234a2a97c6c-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:59.758299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.758138 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3079ecc4-301f-482a-ae74-5234a2a97c6c-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:22:59.996550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:22:59.996517 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v"] Nov 02 14:23:00.005849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:00.005823 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-623b4b/auth-test-model-623b4b-kserve-5f64c84c7c-nxt6v"] Nov 02 14:23:00.529628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:00.529592 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:23:00.529859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:00.529658 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:23:01.099833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:01.099800 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" path="/var/lib/kubelet/pods/3079ecc4-301f-482a-ae74-5234a2a97c6c/volumes" Nov 02 14:23:01.120538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:01.120512 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95"] Nov 02 14:23:01.154706 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:01.154680 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth"] Nov 02 14:23:01.529761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:01.529726 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:23:01.529959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:01.529801 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:23:02.529446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:02.529407 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:23:02.529951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:02.529461 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:23:03.528992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:03.528958 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Nov 02 14:23:03.529177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:03.529018 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Nov 02 14:23:04.575751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:04.575719 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:23:04.576364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:04.576337 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:23:04.577252 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:04.577224 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:23:06.121689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.121630 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" podUID="049be275-7779-40a3-8fc8-bdadb5ffad3c" containerName="payload-processing" containerID="cri-o://a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79" gracePeriod=25 Nov 02 14:23:06.155263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.155215 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" podUID="478cc5a8-20dd-4e7d-b48c-214fab27e94c" containerName="payload-pre-processing" containerID="cri-o://377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9" gracePeriod=25 Nov 02 14:23:06.437852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.437823 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:23:06.467289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.467266 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:23:06.485714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.485689 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049be275-7779-40a3-8fc8-bdadb5ffad3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049be275-7779-40a3-8fc8-bdadb5ffad3c-plugins-config-volume\") pod \"049be275-7779-40a3-8fc8-bdadb5ffad3c\" (UID: \"049be275-7779-40a3-8fc8-bdadb5ffad3c\") " Nov 02 14:23:06.485919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.485739 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049be275-7779-40a3-8fc8-bdadb5ffad3c-kube-api-access-48r77\" (UniqueName: \"kubernetes.io/projected/049be275-7779-40a3-8fc8-bdadb5ffad3c-kube-api-access-48r77\") pod \"049be275-7779-40a3-8fc8-bdadb5ffad3c\" (UID: \"049be275-7779-40a3-8fc8-bdadb5ffad3c\") " Nov 02 14:23:06.486076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.486054 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049be275-7779-40a3-8fc8-bdadb5ffad3c-plugins-config-volume" pod "049be275-7779-40a3-8fc8-bdadb5ffad3c" (UID: "049be275-7779-40a3-8fc8-bdadb5ffad3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:23:06.486244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.486228 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049be275-7779-40a3-8fc8-bdadb5ffad3c-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:06.487997 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.487975 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049be275-7779-40a3-8fc8-bdadb5ffad3c-kube-api-access-48r77" pod "049be275-7779-40a3-8fc8-bdadb5ffad3c" (UID: "049be275-7779-40a3-8fc8-bdadb5ffad3c"). InnerVolumeSpecName "kube-api-access-48r77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:06.586633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.586605 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478cc5a8-20dd-4e7d-b48c-214fab27e94c-kube-api-access-6fz2s\" (UniqueName: \"kubernetes.io/projected/478cc5a8-20dd-4e7d-b48c-214fab27e94c-kube-api-access-6fz2s\") pod \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\" (UID: \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\") " Nov 02 14:23:06.586820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.586672 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478cc5a8-20dd-4e7d-b48c-214fab27e94c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478cc5a8-20dd-4e7d-b48c-214fab27e94c-plugins-config-volume\") pod \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\" (UID: \"478cc5a8-20dd-4e7d-b48c-214fab27e94c\") " Nov 02 14:23:06.587049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.587022 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478cc5a8-20dd-4e7d-b48c-214fab27e94c-plugins-config-volume" pod "478cc5a8-20dd-4e7d-b48c-214fab27e94c" (UID: "478cc5a8-20dd-4e7d-b48c-214fab27e94c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:23:06.587178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.587165 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478cc5a8-20dd-4e7d-b48c-214fab27e94c-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:06.587217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.587181 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48r77\" (UniqueName: \"kubernetes.io/projected/049be275-7779-40a3-8fc8-bdadb5ffad3c-kube-api-access-48r77\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:06.588635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.588616 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478cc5a8-20dd-4e7d-b48c-214fab27e94c-kube-api-access-6fz2s" pod "478cc5a8-20dd-4e7d-b48c-214fab27e94c" (UID: "478cc5a8-20dd-4e7d-b48c-214fab27e94c"). InnerVolumeSpecName "kube-api-access-6fz2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.685901 4431 generic.go:356] "Generic (PLEG): container finished" podID="478cc5a8-20dd-4e7d-b48c-214fab27e94c" containerID="377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9" exitCode=0 Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.686170 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.687933 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" event={"ID":"478cc5a8-20dd-4e7d-b48c-214fab27e94c","Type":"ContainerDied","Data":"377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9"} Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.688063 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth" event={"ID":"478cc5a8-20dd-4e7d-b48c-214fab27e94c","Type":"ContainerDied","Data":"6a00135c46fe61eb1bc4629fb2652fb156ba9c32b31aacde2cae7d78287ca5e2"} Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.688094 4431 scope.go:122] "RemoveContainer" containerID="377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9" Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.689266 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fz2s\" (UniqueName: \"kubernetes.io/projected/478cc5a8-20dd-4e7d-b48c-214fab27e94c-kube-api-access-6fz2s\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.690867 4431 generic.go:356] "Generic (PLEG): container finished" podID="049be275-7779-40a3-8fc8-bdadb5ffad3c" containerID="a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79" exitCode=0 Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.690940 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.691040 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" event={"ID":"049be275-7779-40a3-8fc8-bdadb5ffad3c","Type":"ContainerDied","Data":"a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79"} Nov 02 14:23:06.698177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.691117 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95" event={"ID":"049be275-7779-40a3-8fc8-bdadb5ffad3c","Type":"ContainerDied","Data":"55d1c7ca318684a82ea02aed39e1efd3ef72f6c5933e72bfb98ce3388f109997"} Nov 02 14:23:06.703914 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.703896 4431 scope.go:122] "RemoveContainer" containerID="377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9" Nov 02 14:23:06.704218 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:23:06.704197 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9\": container with ID starting with 377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9 not found: ID does not exist" containerID="377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9" Nov 02 14:23:06.704260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.704224 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9"} err="failed to get container status \"377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9\": rpc error: code = NotFound desc = could not find container \"377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9\": container with ID starting with 377cc296bfe17d384b07c1f9967029d9808b0004b61d6ea990a02eebb6022bd9 not found: ID does not exist" Nov 02 14:23:06.704260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.704241 4431 scope.go:122] "RemoveContainer" containerID="a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79" Nov 02 14:23:06.713250 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.713235 4431 scope.go:122] "RemoveContainer" containerID="a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79" Nov 02 14:23:06.713519 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:23:06.713497 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79\": container with ID starting with a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79 not found: ID does not exist" containerID="a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79" Nov 02 14:23:06.713592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.713528 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79"} err="failed to get container status \"a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79\": rpc error: code = NotFound desc = could not find container \"a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79\": container with ID starting with a8378485cac7bd99bfb5d23a3ec379a8c4a71ef7d63a01d7316a9c21e5befa79 not found: ID does not exist" Nov 02 14:23:06.747394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.747367 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95"] Nov 02 14:23:06.752383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.752358 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-87a1a06b-74cbdbd569-mbn95"] Nov 02 14:23:06.787383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.787365 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth"] Nov 02 14:23:06.794938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:06.794919 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-87a1a06b-765c4f5755-cgxth"] Nov 02 14:23:07.076424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:07.076350 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049be275-7779-40a3-8fc8-bdadb5ffad3c" path="/var/lib/kubelet/pods/049be275-7779-40a3-8fc8-bdadb5ffad3c/volumes" Nov 02 14:23:07.076810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:07.076793 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478cc5a8-20dd-4e7d-b48c-214fab27e94c" path="/var/lib/kubelet/pods/478cc5a8-20dd-4e7d-b48c-214fab27e94c/volumes" Nov 02 14:23:08.831117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:08.831085 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c"] Nov 02 14:23:08.843947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:08.843920 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87a1a06b-8jg8c"] Nov 02 14:23:09.005455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:09.005427 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8"] Nov 02 14:23:09.005699 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:09.005675 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" containerID="cri-o://06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2" gracePeriod=30 Nov 02 14:23:09.076260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:09.076227 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a64636c-8c1b-4f32-aca7-b508c4d17fcf" path="/var/lib/kubelet/pods/4a64636c-8c1b-4f32-aca7-b508c4d17fcf/volumes" Nov 02 14:23:09.821091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:09.821060 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx"] Nov 02 14:23:09.821472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:09.821295 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" containerID="cri-o://0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4" gracePeriod=30 Nov 02 14:23:14.298611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.298589 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:23:14.387841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.387807 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-kube-api-access-ktsdr\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-kube-api-access-ktsdr\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.387927 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62790564-506b-4f53-a9ab-582b2565bc29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62790564-506b-4f53-a9ab-582b2565bc29-istiod-ca-cert\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.387991 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-credential-socket\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388161 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-data\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388257 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-credential-socket" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:14.388458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388382 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62790564-506b-4f53-a9ab-582b2565bc29-istiod-ca-cert" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:23:14.388458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388415 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-data" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:14.388546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388455 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-istio-token\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-istio-token\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388522 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62790564-506b-4f53-a9ab-582b2565bc29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62790564-506b-4f53-a9ab-582b2565bc29-istio-podinfo\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388556 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-certs\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388586 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-socket\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388704 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-envoy\") pod \"62790564-506b-4f53-a9ab-582b2565bc29\" (UID: \"62790564-506b-4f53-a9ab-582b2565bc29\") " Nov 02 14:23:14.388913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.388884 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-certs" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:14.389114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.389090 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-socket" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:14.389493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.389387 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.389493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.389414 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.389493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.389430 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.389493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.389443 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62790564-506b-4f53-a9ab-582b2565bc29-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.389493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.389455 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.391151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.391118 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-istio-token" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:14.391287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.391157 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62790564-506b-4f53-a9ab-582b2565bc29-istio-podinfo" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:23:14.392570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.392548 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-envoy" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:14.392811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.392795 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-kube-api-access-ktsdr" pod "62790564-506b-4f53-a9ab-582b2565bc29" (UID: "62790564-506b-4f53-a9ab-582b2565bc29"). InnerVolumeSpecName "kube-api-access-ktsdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:14.491400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.491354 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62790564-506b-4f53-a9ab-582b2565bc29-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.491400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.491388 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62790564-506b-4f53-a9ab-582b2565bc29-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.491400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.491403 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktsdr\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-kube-api-access-ktsdr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.491698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.491420 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62790564-506b-4f53-a9ab-582b2565bc29-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:14.754556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.754524 4431 generic.go:356] "Generic (PLEG): container finished" podID="62790564-506b-4f53-a9ab-582b2565bc29" containerID="06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2" exitCode=0 Nov 02 14:23:14.754742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.754580 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" Nov 02 14:23:14.754742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.754617 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" event={"ID":"62790564-506b-4f53-a9ab-582b2565bc29","Type":"ContainerDied","Data":"06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2"} Nov 02 14:23:14.754742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.754689 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8" event={"ID":"62790564-506b-4f53-a9ab-582b2565bc29","Type":"ContainerDied","Data":"5a76aa745cbbec80a4f51dcabb1905e932fc10570b82e3c2adc76592ec206281"} Nov 02 14:23:14.754742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.754711 4431 scope.go:122] "RemoveContainer" containerID="06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2" Nov 02 14:23:14.766079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.766062 4431 scope.go:122] "RemoveContainer" containerID="9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5" Nov 02 14:23:14.775614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.775597 4431 scope.go:122] "RemoveContainer" containerID="06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2" Nov 02 14:23:14.775915 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:23:14.775895 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2\": container with ID starting with 06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2 not found: ID does not exist" containerID="06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2" Nov 02 14:23:14.775976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.775923 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2"} err="failed to get container status \"06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2\": rpc error: code = NotFound desc = could not find container \"06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2\": container with ID starting with 06dc518b8f94e841d5d0c181c7eae8e9833ad8d1a0e3bcb8c56f62995e3481b2 not found: ID does not exist" Nov 02 14:23:14.775976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.775940 4431 scope.go:122] "RemoveContainer" containerID="9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5" Nov 02 14:23:14.776204 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:23:14.776189 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5\": container with ID starting with 9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5 not found: ID does not exist" containerID="9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5" Nov 02 14:23:14.776255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.776207 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5"} err="failed to get container status \"9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5\": rpc error: code = NotFound desc = could not find container \"9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5\": container with ID starting with 9e31aa3d90a33a809576b7e71815d7931e7ed1c97ac24470debd10e4a48b15d5 not found: ID does not exist" Nov 02 14:23:14.814091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.814060 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8"] Nov 02 14:23:14.832097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:14.828316 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-87a1a06b-gw-openshift-default-bd976cdf7-7bkw8"] Nov 02 14:23:15.078434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.078404 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62790564-506b-4f53-a9ab-582b2565bc29" path="/var/lib/kubelet/pods/62790564-506b-4f53-a9ab-582b2565bc29/volumes" Nov 02 14:23:15.100897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.100873 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209259 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-credential-socket\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209363 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-istio-token\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-istio-token\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209400 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-envoy\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209431 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-kube-api-access-65lxd\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-kube-api-access-65lxd\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209495 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-certs\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209535 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-socket\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209574 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4788beeb-e260-499f-b1fe-1c5198de154e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4788beeb-e260-499f-b1fe-1c5198de154e-istiod-ca-cert\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209606 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4788beeb-e260-499f-b1fe-1c5198de154e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4788beeb-e260-499f-b1fe-1c5198de154e-istio-podinfo\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.222804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.209766 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-data\") pod \"4788beeb-e260-499f-b1fe-1c5198de154e\" (UID: \"4788beeb-e260-499f-b1fe-1c5198de154e\") " Nov 02 14:23:15.246799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.226081 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4788beeb-e260-499f-b1fe-1c5198de154e-istiod-ca-cert" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:23:15.246799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.226382 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-socket" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:15.246799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.232300 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.246799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.232329 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4788beeb-e260-499f-b1fe-1c5198de154e-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.246799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.232855 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-data" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:15.246799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.236195 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-kube-api-access-65lxd" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "kube-api-access-65lxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:15.247322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.246859 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-certs" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:15.260264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.254247 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-istio-token" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:23:15.260264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.254378 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-envoy" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:15.260264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.255901 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-credential-socket" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:23:15.270486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.269000 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4788beeb-e260-499f-b1fe-1c5198de154e-istio-podinfo" pod "4788beeb-e260-499f-b1fe-1c5198de154e" (UID: "4788beeb-e260-499f-b1fe-1c5198de154e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:23:15.333584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.333501 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.333584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.333532 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.333584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.333547 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65lxd\" (UniqueName: \"kubernetes.io/projected/4788beeb-e260-499f-b1fe-1c5198de154e-kube-api-access-65lxd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.333584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.333557 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.333584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.333566 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4788beeb-e260-499f-b1fe-1c5198de154e-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.333584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.333574 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.333584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.333581 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4788beeb-e260-499f-b1fe-1c5198de154e-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:23:15.763533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.763497 4431 generic.go:356] "Generic (PLEG): container finished" podID="4788beeb-e260-499f-b1fe-1c5198de154e" containerID="0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4" exitCode=0 Nov 02 14:23:15.763714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.763547 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" Nov 02 14:23:15.763714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.763579 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" event={"ID":"4788beeb-e260-499f-b1fe-1c5198de154e","Type":"ContainerDied","Data":"0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4"} Nov 02 14:23:15.763714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.763637 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx" event={"ID":"4788beeb-e260-499f-b1fe-1c5198de154e","Type":"ContainerDied","Data":"8fd3632305f16d1d98137a7c1fca69ef3356c4333ce4c804902e6210c3b98974"} Nov 02 14:23:15.763714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.763662 4431 scope.go:122] "RemoveContainer" containerID="0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4" Nov 02 14:23:15.774421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.774391 4431 scope.go:122] "RemoveContainer" containerID="0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4" Nov 02 14:23:15.774685 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:23:15.774665 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4\": container with ID starting with 0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4 not found: ID does not exist" containerID="0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4" Nov 02 14:23:15.774750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.774696 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4"} err="failed to get container status \"0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4\": rpc error: code = NotFound desc = could not find container \"0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4\": container with ID starting with 0d78d385bcc9ae8dae8ade3eb7a868c9fcd79a48122e3bfe7f9ee102d91170d4 not found: ID does not exist" Nov 02 14:23:15.822096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.822074 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx"] Nov 02 14:23:15.834592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:15.834573 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-wt9kx"] Nov 02 14:23:16.034652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:16.034559 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4"] Nov 02 14:23:16.040019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:16.039998 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd44703-lb5w4"] Nov 02 14:23:16.704012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:16.703973 4431 scope.go:122] "RemoveContainer" containerID="40540e9241d678f8f4b0a12e1dffb71404dc8d5a797b8b90111225bbada4eaff" Nov 02 14:23:16.721139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:16.718856 4431 scope.go:122] "RemoveContainer" containerID="12c42f377e5c75dc191fd547186ced9c34010e84956fcceb0966ab8a7e51e1ad" Nov 02 14:23:16.730239 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:16.730216 4431 scope.go:122] "RemoveContainer" containerID="8128b561fa1fabfd8a710ffc744153e07dc5ad8c5ece71556d3079a2090c3a30" Nov 02 14:23:16.739456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:16.739432 4431 scope.go:122] "RemoveContainer" containerID="673bc71c2a70e7b428c092d151cd20ef40d1204c142e95e46bddec021c7fe066" Nov 02 14:23:16.747890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:16.747869 4431 scope.go:122] "RemoveContainer" containerID="c3a7dfd38db0c573ebd0eb9ef9efc297afbe0f83d3764bb47536e522e6e67144" Nov 02 14:23:17.080627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:17.080543 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f65433-b54b-4f98-bcd7-6e3bd65d624a" path="/var/lib/kubelet/pods/25f65433-b54b-4f98-bcd7-6e3bd65d624a/volumes" Nov 02 14:23:17.081619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:17.081595 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" path="/var/lib/kubelet/pods/4788beeb-e260-499f-b1fe-1c5198de154e/volumes" Nov 02 14:23:19.624203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:19.624170 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:23:21.302995 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:23:21.302951 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:23:27.032385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:27.032352 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8"] Nov 02 14:23:27.045139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:27.045113 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12879879-47mg8"] Nov 02 14:23:27.075197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:27.075169 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cf3032-cebe-423b-8212-5bc7512c3462" path="/var/lib/kubelet/pods/02cf3032-cebe-423b-8212-5bc7512c3462/volumes" Nov 02 14:23:42.962855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:42.958874 4431 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Nov 02 14:23:43.184912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:43.184864 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:43.296855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:43.295883 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:43.297045 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:43.296948 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": context deadline exceeded" Nov 02 14:23:43.380802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:43.374857 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:43.422910 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:43.422864 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:23:43.522163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:43.522114 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:44.060355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.060312 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs"] Nov 02 14:23:44.086798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.085156 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d89100b-jqdjs"] Nov 02 14:23:44.873803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.869918 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn"] Nov 02 14:23:44.873803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.870711 4431 cpu_manager.go:414] "containerMap: removing container" podUID="049be275-7779-40a3-8fc8-bdadb5ffad3c" containerName="payload-processing" Nov 02 14:23:44.873803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.870730 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049be275-7779-40a3-8fc8-bdadb5ffad3c" containerName="payload-processing" Nov 02 14:23:44.873803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.870755 4431 cpu_manager.go:414] "containerMap: removing container" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" Nov 02 14:23:44.873803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.870762 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.876829 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.876853 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.876908 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerName="main" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.876916 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerName="main" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.876935 4431 cpu_manager.go:414] "containerMap: removing container" podUID="478cc5a8-20dd-4e7d-b48c-214fab27e94c" containerName="payload-pre-processing" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.876943 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478cc5a8-20dd-4e7d-b48c-214fab27e94c" containerName="payload-pre-processing" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.877112 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4788beeb-e260-499f-b1fe-1c5198de154e" containerName="istio-proxy" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.877131 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3079ecc4-301f-482a-ae74-5234a2a97c6c" containerName="main" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.877143 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="478cc5a8-20dd-4e7d-b48c-214fab27e94c" containerName="payload-pre-processing" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.877153 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.877165 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="049be275-7779-40a3-8fc8-bdadb5ffad3c" containerName="payload-processing" Nov 02 14:23:44.880046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.877177 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" Nov 02 14:23:44.882064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.882041 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:44.902530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.902505 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-673ba8-openshift-default-dockercfg-g689j\"" Nov 02 14:23:44.921205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:44.921169 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn"] Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.019649 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj4xw\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-kube-api-access-tj4xw\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.019726 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-envoy\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.019763 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istiod-ca-cert\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.019810 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-certs\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.019870 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-socket\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.019904 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-data\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.019945 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-token\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.019992 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-podinfo\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.021798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.020608 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-credential-socket\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.122576 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-token\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.122641 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-podinfo\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.122754 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-credential-socket\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.122817 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj4xw\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-kube-api-access-tj4xw\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.122874 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-envoy\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.122908 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istiod-ca-cert\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.122942 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-certs\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.123006 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-socket\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.123044 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-data\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.123492 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-data\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.125099 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-certs\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.125755 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istiod-ca-cert\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.126023 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-credential-socket\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.140636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.127448 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-socket\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.182052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.182017 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3656a8-c36f-496e-8898-49a6ff62837f" path="/var/lib/kubelet/pods/6a3656a8-c36f-496e-8898-49a6ff62837f/volumes" Nov 02 14:23:45.195406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.195373 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-token\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.198615 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.198535 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj4xw\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-kube-api-access-tj4xw\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.210228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.209735 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-envoy\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.221797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.217375 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-podinfo\") pod \"e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.515921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.515016 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:45.849013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.848937 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-54cc564d84-2md8z"] Nov 02 14:23:45.850089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.850062 4431 cpu_manager.go:414] "containerMap: removing container" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" Nov 02 14:23:45.850256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.850244 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62790564-506b-4f53-a9ab-582b2565bc29" containerName="istio-proxy" Nov 02 14:23:45.902151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.902122 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cc564d84-2md8z"] Nov 02 14:23:45.909796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.902625 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:45.926642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.926619 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 14:23:45.939761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.939741 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fa41858d6534\"" Nov 02 14:23:45.939917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.939756 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 14:23:45.939917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.939787 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 14:23:45.940032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.939792 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 14:23:45.940032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:45.939827 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-56zdx\"" Nov 02 14:23:46.006233 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.006208 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn"] Nov 02 14:23:46.008093 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:23:46.008069 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d74219_9bc6_492c_b1fb_01a2c2b22ebf.slice/crio-625be102d779e91d65c5695eadb5436d3b3ce9e5b2d39efd49298b3c82cb11b6 WatchSource:0}: Error finding container 625be102d779e91d65c5695eadb5436d3b3ce9e5b2d39efd49298b3c82cb11b6: Status 404 returned error can't find the container with id 625be102d779e91d65c5695eadb5436d3b3ce9e5b2d39efd49298b3c82cb11b6 Nov 02 14:23:46.010965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.010558 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:23:46.010965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.010619 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:23:46.010965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.010648 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:23:46.105856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.105816 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-plugins-config-volume\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.106006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.105900 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-routing-overlay-volume\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.106078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.106025 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtgg\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-kube-api-access-9wtgg\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.106140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.106123 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-provider-credentials\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.112849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.112812 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" event={"ID":"80d74219-9bc6-492c-b1fb-01a2c2b22ebf","Type":"ContainerStarted","Data":"665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212"} Nov 02 14:23:46.113067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.113041 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" event={"ID":"80d74219-9bc6-492c-b1fb-01a2c2b22ebf","Type":"ContainerStarted","Data":"625be102d779e91d65c5695eadb5436d3b3ce9e5b2d39efd49298b3c82cb11b6"} Nov 02 14:23:46.163626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.163561 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" podStartSLOduration=2.163526034 podStartE2EDuration="2.163526034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:46.160565363 +0000 UTC m=+1591.197670288" watchObservedRunningTime="2026-11-02 14:23:46.163526034 +0000 UTC m=+1591.200630966" Nov 02 14:23:46.207164 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.207127 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtgg\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-kube-api-access-9wtgg\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.207368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.207192 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-provider-credentials\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.207368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.207241 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-plugins-config-volume\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.207368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.207282 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-routing-overlay-volume\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.207923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.207897 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-routing-overlay-volume\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.208545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.208525 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-plugins-config-volume\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.210864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.210841 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-provider-credentials\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.233858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.233830 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtgg\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-kube-api-access-9wtgg\") pod \"payload-processing-external-model-54cc564d84-2md8z\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.516640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.516611 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:46.531096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.531064 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:46.860865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:46.860837 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cc564d84-2md8z"] Nov 02 14:23:46.861276 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:23:46.861244 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70385f83_e53d_43d2_80e3_9fc9d8859768.slice/crio-d612e241e0f96fbce2c5c2a72514a4ff4b32f1711842c5c0c9c8b4f31ab32e9f WatchSource:0}: Error finding container d612e241e0f96fbce2c5c2a72514a4ff4b32f1711842c5c0c9c8b4f31ab32e9f: Status 404 returned error can't find the container with id d612e241e0f96fbce2c5c2a72514a4ff4b32f1711842c5c0c9c8b4f31ab32e9f Nov 02 14:23:47.122041 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:47.121943 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" event={"ID":"70385f83-e53d-43d2-80e3-9fc9d8859768","Type":"ContainerStarted","Data":"99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2"} Nov 02 14:23:47.122041 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:47.122008 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" event={"ID":"70385f83-e53d-43d2-80e3-9fc9d8859768","Type":"ContainerStarted","Data":"d612e241e0f96fbce2c5c2a72514a4ff4b32f1711842c5c0c9c8b4f31ab32e9f"} Nov 02 14:23:47.122041 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:47.122040 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:47.198999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:47.198946 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" podStartSLOduration=2.198928839 podStartE2EDuration="2.198928839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:47.192106163 +0000 UTC m=+1592.229211097" watchObservedRunningTime="2026-11-02 14:23:47.198928839 +0000 UTC m=+1592.236033774" Nov 02 14:23:47.522996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:47.522969 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:48.130606 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:48.130571 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:48.131425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:48.131407 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:23:50.891386 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:50.891339 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": read tcp 10.133.0.2:36124->10.133.0.154:15021: read: connection reset by peer" start-of-body= Nov 02 14:23:50.892003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:50.891412 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": read tcp 10.133.0.2:36124->10.133.0.154:15021: read: connection reset by peer" Nov 02 14:23:51.152999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:51.152936 4431 generic.go:356] "Generic (PLEG): container finished" podID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerID="da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba" exitCode=0 Nov 02 14:23:51.153142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:51.153009 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" event={"ID":"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0","Type":"ContainerDied","Data":"da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba"} Nov 02 14:23:51.153364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:51.153349 4431 scope.go:122] "RemoveContainer" containerID="da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba" Nov 02 14:23:51.154671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:51.154644 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:23:51.154730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:51.154707 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:23:51.154766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:51.154738 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:23:51.289503 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:23:51.289464 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:23:52.161997 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:52.161964 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" event={"ID":"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0","Type":"ContainerStarted","Data":"d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e"} Nov 02 14:23:53.087806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:53.083160 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 02 14:23:53.087806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:53.083217 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 02 14:23:53.097280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:53.097253 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:23:54.088802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:54.080117 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 02 14:23:54.088802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:54.080171 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 02 14:23:55.095841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:55.095809 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 02 14:23:55.096340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:55.096302 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 02 14:23:55.344931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:55.344893 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:23:56.082196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:56.082159 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 02 14:23:56.082477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:56.082445 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 02 14:23:57.082921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:57.082883 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 02 14:23:57.083552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:57.083519 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 02 14:23:58.066900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:58.066837 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb"] Nov 02 14:23:58.086800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:58.080418 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8ef107-n56pb"] Nov 02 14:23:58.160800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:58.145977 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:23:58.295847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:58.283128 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:23:58.316844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:58.316063 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:23:58.320054 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:58.319986 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:23:59.399801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:59.379421 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ad3e28-230b-4cfe-a186-d8afcf0e1320" path="/var/lib/kubelet/pods/27ad3e28-230b-4cfe-a186-d8afcf0e1320/volumes" Nov 02 14:23:59.401937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:59.382238 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:59.402392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:59.389372 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:59.410805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:59.404818 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw"] Nov 02 14:23:59.410805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:59.404866 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6eeee79-29wvw"] Nov 02 14:23:59.493896 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:59.493858 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:23:59.558905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:23:59.558864 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:24:01.076153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:01.076120 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3faa43eb-60a8-4730-aee4-817f2feff6a4" path="/var/lib/kubelet/pods/3faa43eb-60a8-4730-aee4-817f2feff6a4/volumes" Nov 02 14:24:05.348580 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.348549 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs"] Nov 02 14:24:05.352406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.352390 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.382333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.382303 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:24:05.382333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.382323 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:24:05.387888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.387861 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs"] Nov 02 14:24:05.435263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.435237 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.435436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.435286 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.435436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.435346 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqxt\" (UniqueName: \"kubernetes.io/projected/9d3009a4-1166-4c47-825e-9b62544dfc46-kube-api-access-gsqxt\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.536841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.536802 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.537075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.536880 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.537075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.536931 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqxt\" (UniqueName: \"kubernetes.io/projected/9d3009a4-1166-4c47-825e-9b62544dfc46-kube-api-access-gsqxt\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.538294 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.538208 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.540090 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.540067 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.563689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.563656 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqxt\" (UniqueName: \"kubernetes.io/projected/9d3009a4-1166-4c47-825e-9b62544dfc46-kube-api-access-gsqxt\") pod \"maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.665542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.665517 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:05.965756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:05.965731 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs"] Nov 02 14:24:05.966893 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:24:05.966863 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3009a4_1166_4c47_825e_9b62544dfc46.slice/crio-b84b27ab48ab38a4a54b59d2e26fd6279e12aafc446b5c2bded8522012899a20 WatchSource:0}: Error finding container b84b27ab48ab38a4a54b59d2e26fd6279e12aafc446b5c2bded8522012899a20: Status 404 returned error can't find the container with id b84b27ab48ab38a4a54b59d2e26fd6279e12aafc446b5c2bded8522012899a20 Nov 02 14:24:06.429196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:06.429156 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" event={"ID":"9d3009a4-1166-4c47-825e-9b62544dfc46","Type":"ContainerStarted","Data":"a0386251bc3c2cbf67167929c0584ee476919476eefa453a2cd5341254a60bca"} Nov 02 14:24:06.429587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:06.429223 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" event={"ID":"9d3009a4-1166-4c47-825e-9b62544dfc46","Type":"ContainerStarted","Data":"b84b27ab48ab38a4a54b59d2e26fd6279e12aafc446b5c2bded8522012899a20"} Nov 02 14:24:06.476056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:06.476003 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" podStartSLOduration=1.4759866750000001 podStartE2EDuration="1.475986675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:06.472717853 +0000 UTC m=+1611.509822785" watchObservedRunningTime="2026-11-02 14:24:06.475986675 +0000 UTC m=+1611.513091608" Nov 02 14:24:07.440551 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:07.440497 4431 generic.go:356] "Generic (PLEG): container finished" podID="9d3009a4-1166-4c47-825e-9b62544dfc46" containerID="a0386251bc3c2cbf67167929c0584ee476919476eefa453a2cd5341254a60bca" exitCode=0 Nov 02 14:24:07.441184 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:07.440615 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" event={"ID":"9d3009a4-1166-4c47-825e-9b62544dfc46","Type":"ContainerDied","Data":"a0386251bc3c2cbf67167929c0584ee476919476eefa453a2cd5341254a60bca"} Nov 02 14:24:08.602601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.602579 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:08.776844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.776813 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3009a4-1166-4c47-825e-9b62544dfc46-kube-api-access-gsqxt\" (UniqueName: \"kubernetes.io/projected/9d3009a4-1166-4c47-825e-9b62544dfc46-kube-api-access-gsqxt\") pod \"9d3009a4-1166-4c47-825e-9b62544dfc46\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " Nov 02 14:24:08.776998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.776882 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-openshift-ca\") pod \"9d3009a4-1166-4c47-825e-9b62544dfc46\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " Nov 02 14:24:08.776998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.776903 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-xks-ca\") pod \"9d3009a4-1166-4c47-825e-9b62544dfc46\" (UID: \"9d3009a4-1166-4c47-825e-9b62544dfc46\") " Nov 02 14:24:08.777303 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.777267 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-openshift-ca" pod "9d3009a4-1166-4c47-825e-9b62544dfc46" (UID: "9d3009a4-1166-4c47-825e-9b62544dfc46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:08.779078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.779034 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-xks-ca" pod "9d3009a4-1166-4c47-825e-9b62544dfc46" (UID: "9d3009a4-1166-4c47-825e-9b62544dfc46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:24:08.779211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.779131 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3009a4-1166-4c47-825e-9b62544dfc46-kube-api-access-gsqxt" pod "9d3009a4-1166-4c47-825e-9b62544dfc46" (UID: "9d3009a4-1166-4c47-825e-9b62544dfc46"). InnerVolumeSpecName "kube-api-access-gsqxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:08.878100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.878024 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqxt\" (UniqueName: \"kubernetes.io/projected/9d3009a4-1166-4c47-825e-9b62544dfc46-kube-api-access-gsqxt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:08.878100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.878049 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:08.878100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:08.878059 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d3009a4-1166-4c47-825e-9b62544dfc46-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:09.454942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:09.454912 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" Nov 02 14:24:09.455113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:09.454951 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs" event={"ID":"9d3009a4-1166-4c47-825e-9b62544dfc46","Type":"ContainerDied","Data":"b84b27ab48ab38a4a54b59d2e26fd6279e12aafc446b5c2bded8522012899a20"} Nov 02 14:24:09.455113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:09.454998 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b84b27ab48ab38a4a54b59d2e26fd6279e12aafc446b5c2bded8522012899a20" Nov 02 14:24:11.037276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:11.037244 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f"] Nov 02 14:24:11.048953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:11.048923 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4d9d31-rq57f"] Nov 02 14:24:11.087766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:11.087731 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972" path="/var/lib/kubelet/pods/65b1c0e7-2a7e-4a1d-b9b4-9b05d1a44972/volumes" Nov 02 14:24:16.846752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:16.846715 4431 scope.go:122] "RemoveContainer" containerID="ecb5fe7f39b9cf0e7facbea1b6194b6e8ead6b9cc8755377b281a8b92841b577" Nov 02 14:24:16.861507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:16.861482 4431 scope.go:122] "RemoveContainer" containerID="11775779dcfb3e14cdd7c1df00af494358e3b1d7f62a238b8af6398e03c6ee47" Nov 02 14:24:16.876126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:16.876099 4431 scope.go:122] "RemoveContainer" containerID="4c8add156fc79d688d49c62711a1f5409ab58df3aee1f7575f884ec083b2bb23" Nov 02 14:24:21.283851 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:24:21.283815 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:24:22.033542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:22.033508 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz"] Nov 02 14:24:22.046459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:22.046425 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6addd867-8kbzz"] Nov 02 14:24:22.242643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:22.242613 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cc564d84-2md8z"] Nov 02 14:24:23.079089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:23.079049 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e" path="/var/lib/kubelet/pods/6bcd2d3a-e64f-43fe-8e5f-ea6eba61608e/volumes" Nov 02 14:24:27.243119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.243051 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" podUID="70385f83-e53d-43d2-80e3-9fc9d8859768" containerName="payload-processing" containerID="cri-o://99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2" gracePeriod=25 Nov 02 14:24:27.501121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.501069 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:24:27.601854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.601814 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-routing-overlay-volume\") pod \"70385f83-e53d-43d2-80e3-9fc9d8859768\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " Nov 02 14:24:27.601997 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.601907 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-provider-credentials\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-provider-credentials\") pod \"70385f83-e53d-43d2-80e3-9fc9d8859768\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " Nov 02 14:24:27.601997 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.601930 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-kube-api-access-9wtgg\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-kube-api-access-9wtgg\") pod \"70385f83-e53d-43d2-80e3-9fc9d8859768\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " Nov 02 14:24:27.602087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.602034 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-plugins-config-volume\") pod \"70385f83-e53d-43d2-80e3-9fc9d8859768\" (UID: \"70385f83-e53d-43d2-80e3-9fc9d8859768\") " Nov 02 14:24:27.602174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.602148 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-routing-overlay-volume" pod "70385f83-e53d-43d2-80e3-9fc9d8859768" (UID: "70385f83-e53d-43d2-80e3-9fc9d8859768"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:27.602476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.602449 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-plugins-config-volume" pod "70385f83-e53d-43d2-80e3-9fc9d8859768" (UID: "70385f83-e53d-43d2-80e3-9fc9d8859768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:27.602608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.602588 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:27.602673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.602617 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70385f83-e53d-43d2-80e3-9fc9d8859768-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:27.603988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.603967 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-kube-api-access-9wtgg" pod "70385f83-e53d-43d2-80e3-9fc9d8859768" (UID: "70385f83-e53d-43d2-80e3-9fc9d8859768"). InnerVolumeSpecName "kube-api-access-9wtgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:27.604071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.604032 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-provider-credentials" pod "70385f83-e53d-43d2-80e3-9fc9d8859768" (UID: "70385f83-e53d-43d2-80e3-9fc9d8859768"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:27.636511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.636484 4431 generic.go:356] "Generic (PLEG): container finished" podID="70385f83-e53d-43d2-80e3-9fc9d8859768" containerID="99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2" exitCode=0 Nov 02 14:24:27.636643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.636544 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" Nov 02 14:24:27.636643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.636568 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" event={"ID":"70385f83-e53d-43d2-80e3-9fc9d8859768","Type":"ContainerDied","Data":"99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2"} Nov 02 14:24:27.636643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.636631 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54cc564d84-2md8z" event={"ID":"70385f83-e53d-43d2-80e3-9fc9d8859768","Type":"ContainerDied","Data":"d612e241e0f96fbce2c5c2a72514a4ff4b32f1711842c5c0c9c8b4f31ab32e9f"} Nov 02 14:24:27.636830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.636661 4431 scope.go:122] "RemoveContainer" containerID="99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2" Nov 02 14:24:27.646444 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.646424 4431 scope.go:122] "RemoveContainer" containerID="99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2" Nov 02 14:24:27.646692 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:24:27.646672 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2\": container with ID starting with 99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2 not found: ID does not exist" containerID="99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2" Nov 02 14:24:27.646743 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.646700 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2"} err="failed to get container status \"99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2\": rpc error: code = NotFound desc = could not find container \"99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2\": container with ID starting with 99a9c251e6ef1a5c495357d84a7bf7786602f079868040df4543303a7836bac2 not found: ID does not exist" Nov 02 14:24:27.694148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.694117 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cc564d84-2md8z"] Nov 02 14:24:27.700858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.700831 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-54cc564d84-2md8z"] Nov 02 14:24:27.704242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.704223 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:27.704312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:27.704245 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wtgg\" (UniqueName: \"kubernetes.io/projected/70385f83-e53d-43d2-80e3-9fc9d8859768-kube-api-access-9wtgg\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:28.711202 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:28.711172 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs"] Nov 02 14:24:28.723891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:28.723864 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8416e2e2-5pzbs"] Nov 02 14:24:29.077131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:29.077047 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70385f83-e53d-43d2-80e3-9fc9d8859768" path="/var/lib/kubelet/pods/70385f83-e53d-43d2-80e3-9fc9d8859768/volumes" Nov 02 14:24:29.077573 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:29.077553 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3009a4-1166-4c47-825e-9b62544dfc46" path="/var/lib/kubelet/pods/9d3009a4-1166-4c47-825e-9b62544dfc46/volumes" Nov 02 14:24:31.505593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.505394 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r"] Nov 02 14:24:31.509133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.509105 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9d3009a4-1166-4c47-825e-9b62544dfc46" containerName="revoke-keys" Nov 02 14:24:31.509133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.509134 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3009a4-1166-4c47-825e-9b62544dfc46" containerName="revoke-keys" Nov 02 14:24:31.509291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.509161 4431 cpu_manager.go:414] "containerMap: removing container" podUID="70385f83-e53d-43d2-80e3-9fc9d8859768" containerName="payload-processing" Nov 02 14:24:31.509291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.509169 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70385f83-e53d-43d2-80e3-9fc9d8859768" containerName="payload-processing" Nov 02 14:24:31.509362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.509350 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3009a4-1166-4c47-825e-9b62544dfc46" containerName="revoke-keys" Nov 02 14:24:31.509398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.509365 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="70385f83-e53d-43d2-80e3-9fc9d8859768" containerName="payload-processing" Nov 02 14:24:31.515362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.515338 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.538023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.537998 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b3690233-openshift-default-dockercfg-g2rmw\"" Nov 02 14:24:31.538177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.538159 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r"] Nov 02 14:24:31.672912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.672869 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-certs\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.673281 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.673257 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-socket\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.673529 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.673511 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-data\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.673700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.673682 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-envoy\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.673915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.673895 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2c7\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-kube-api-access-hg2c7\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.674100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.674084 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181db062-305e-4f1f-942c-3b1feed06efe-istio-podinfo\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.674414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.674374 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-credential-socket\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.674962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.674943 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-istio-token\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.675149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.675131 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181db062-305e-4f1f-942c-3b1feed06efe-istiod-ca-cert\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.776677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.776570 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-certs\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.777737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.777710 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-socket\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.778007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.777987 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-data\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.778140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.778126 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-envoy\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.778273 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.778256 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2c7\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-kube-api-access-hg2c7\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.778402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.778389 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181db062-305e-4f1f-942c-3b1feed06efe-istio-podinfo\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.778651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.778637 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-credential-socket\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.789167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.789146 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-istio-token\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.789336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.789319 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181db062-305e-4f1f-942c-3b1feed06efe-istiod-ca-cert\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.789528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.788418 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-data\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.789626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.777309 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-certs\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.789970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.788180 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-socket\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.790681 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.790658 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181db062-305e-4f1f-942c-3b1feed06efe-istiod-ca-cert\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.796829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.791071 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-credential-socket\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.796829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.794191 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-envoy\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.807871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.807822 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181db062-305e-4f1f-942c-3b1feed06efe-istio-podinfo\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.820236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.820206 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-istio-token\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.828484 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.828459 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2c7\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-kube-api-access-hg2c7\") pod \"e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:31.836045 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:31.836018 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:32.158865 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:24:32.158837 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181db062_305e_4f1f_942c_3b1feed06efe.slice/crio-b6b9e3ecdf806664ba4013cdb7b25f6314bc86a2d858cc60ed640795df17fc79 WatchSource:0}: Error finding container b6b9e3ecdf806664ba4013cdb7b25f6314bc86a2d858cc60ed640795df17fc79: Status 404 returned error can't find the container with id b6b9e3ecdf806664ba4013cdb7b25f6314bc86a2d858cc60ed640795df17fc79 Nov 02 14:24:32.159859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:32.159831 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r"] Nov 02 14:24:32.161440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:32.161406 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:24:32.161530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:32.161471 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:24:32.161530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:32.161502 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:24:32.674237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:32.674203 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" event={"ID":"181db062-305e-4f1f-942c-3b1feed06efe","Type":"ContainerStarted","Data":"fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2"} Nov 02 14:24:32.674610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:32.674254 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" event={"ID":"181db062-305e-4f1f-942c-3b1feed06efe","Type":"ContainerStarted","Data":"b6b9e3ecdf806664ba4013cdb7b25f6314bc86a2d858cc60ed640795df17fc79"} Nov 02 14:24:32.740213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:32.740166 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podStartSLOduration=1.740152892 podStartE2EDuration="1.740152892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:32.736912827 +0000 UTC m=+1637.774017759" watchObservedRunningTime="2026-11-02 14:24:32.740152892 +0000 UTC m=+1637.777257825" Nov 02 14:24:32.836317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:32.836290 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:33.842188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:33.842162 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:34.690887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:34.690855 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:34.692185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:34.692154 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:24:37.031974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:37.031932 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv"] Nov 02 14:24:37.044984 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:37.044953 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cde6729-klhdv"] Nov 02 14:24:37.075856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:37.075832 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c851ac2e-f840-4b78-bd26-85a3653ff657" path="/var/lib/kubelet/pods/c851ac2e-f840-4b78-bd26-85a3653ff657/volumes" Nov 02 14:24:38.571708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:38.571674 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:24:42.824824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:42.824789 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb"] Nov 02 14:24:42.825261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:42.825013 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" containerID="cri-o://0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82" gracePeriod=30 Nov 02 14:24:42.838048 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:42.838022 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt"] Nov 02 14:24:42.838236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:42.838208 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" containerID="cri-o://fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243" gracePeriod=30 Nov 02 14:24:42.840974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:42.840953 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8"] Nov 02 14:24:42.841148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:42.841129 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" containerID="cri-o://e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13" gracePeriod=30 Nov 02 14:24:48.145959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.145722 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:24:48.198349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.198317 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:24:48.206945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.206914 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-certs\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.207092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.206969 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-kube-api-access-cffmp\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-kube-api-access-cffmp\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.207092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.207042 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istiod-ca-cert\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.207194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.207177 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-credential-socket\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.207231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.207212 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-data\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.207289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.207276 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-token\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-token\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.207324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.207312 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-podinfo\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.207444 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.207425 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-envoy\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.207546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.207530 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-socket\") pod \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\" (UID: \"2f32b447-e76e-42bc-a8fb-61a7ba9e9898\") " Nov 02 14:24:48.210346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.209181 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-data" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.210346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.209510 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istiod-ca-cert" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:48.210346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.209695 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-certs" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.210346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.209816 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-socket" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.210346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.209911 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-credential-socket" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.234746 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.234694 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-token" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:48.240297 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.240265 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-kube-api-access-cffmp" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "kube-api-access-cffmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:48.240297 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.240269 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-envoy" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.240491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.240292 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-podinfo" pod "2f32b447-e76e-42bc-a8fb-61a7ba9e9898" (UID: "2f32b447-e76e-42bc-a8fb-61a7ba9e9898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:24:48.252601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.252578 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:24:48.308641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308615 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-kube-api-access-9b8cn\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-kube-api-access-9b8cn\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.308833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308706 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-certs\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.308833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308740 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-kube-api-access-666wv\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-kube-api-access-666wv\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.308833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308765 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3052f4d4-e528-4e62-8b00-b48e23ce005a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3052f4d4-e528-4e62-8b00-b48e23ce005a-istiod-ca-cert\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.308833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308814 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-envoy\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.309077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308857 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-certs\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.309077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308879 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-data\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.309077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308903 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-socket\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.309077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308956 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istiod-ca-cert\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.309077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.308990 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-token\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-token\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.309077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309008 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-token\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-token\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.309077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309026 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-socket\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.309077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309068 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-podinfo\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.309487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309099 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-credential-socket\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.309487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309127 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-data\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.309487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309165 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-envoy\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.309487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309181 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-podinfo\") pod \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\" (UID: \"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4\") " Nov 02 14:24:48.309487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309205 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-credential-socket\") pod \"3052f4d4-e528-4e62-8b00-b48e23ce005a\" (UID: \"3052f4d4-e528-4e62-8b00-b48e23ce005a\") " Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309584 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309599 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309613 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309626 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309636 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309644 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309652 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309660 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffmp\" (UniqueName: \"kubernetes.io/projected/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-kube-api-access-cffmp\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.309741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309668 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f32b447-e76e-42bc-a8fb-61a7ba9e9898-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.310220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.309901 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-credential-socket" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.310802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.310537 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istiod-ca-cert" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:48.311164 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.310978 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-certs" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.311164 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.311012 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-credential-socket" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.311164 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.311153 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-data" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.311479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.311418 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-socket" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.311561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.311504 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-socket" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.311988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.311759 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-data" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.311988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.311811 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-certs" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.312906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.312878 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3052f4d4-e528-4e62-8b00-b48e23ce005a-istiod-ca-cert" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:24:48.313012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.312992 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-kube-api-access-9b8cn" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "kube-api-access-9b8cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:48.314967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.314927 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-token" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:48.315113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.315092 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-envoy" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.315204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.315139 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-podinfo" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:24:48.315536 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.315506 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-envoy" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:24:48.316005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.315981 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-kube-api-access-666wv" pod "3052f4d4-e528-4e62-8b00-b48e23ce005a" (UID: "3052f4d4-e528-4e62-8b00-b48e23ce005a"). InnerVolumeSpecName "kube-api-access-666wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:48.316233 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.316209 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-token" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:24:48.316646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.316617 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-podinfo" pod "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" (UID: "74ccc3a9-7f72-467c-a8dd-b4783c6c08f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:24:48.411344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411314 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411341 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411350 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411359 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411367 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411375 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411383 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411392 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411399 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411407 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411414 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b8cn\" (UniqueName: \"kubernetes.io/projected/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-kube-api-access-9b8cn\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411424 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3052f4d4-e528-4e62-8b00-b48e23ce005a-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411432 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-666wv\" (UniqueName: \"kubernetes.io/projected/3052f4d4-e528-4e62-8b00-b48e23ce005a-kube-api-access-666wv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411439 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3052f4d4-e528-4e62-8b00-b48e23ce005a-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411447 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411455 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411462 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.411562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.411470 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:24:48.689196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.689157 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": read tcp 10.133.0.2:53468->10.133.0.157:15021: read: connection reset by peer" start-of-body= Nov 02 14:24:48.689513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.689234 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": read tcp 10.133.0.2:53468->10.133.0.157:15021: read: connection reset by peer" Nov 02 14:24:48.805222 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.805193 4431 generic.go:356] "Generic (PLEG): container finished" podID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerID="665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212" exitCode=0 Nov 02 14:24:48.805390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.805261 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" event={"ID":"80d74219-9bc6-492c-b1fb-01a2c2b22ebf","Type":"ContainerDied","Data":"665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212"} Nov 02 14:24:48.805729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.805705 4431 scope.go:122] "RemoveContainer" containerID="665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212" Nov 02 14:24:48.806912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.806886 4431 generic.go:356] "Generic (PLEG): container finished" podID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerID="0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82" exitCode=0 Nov 02 14:24:48.807033 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.806941 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" Nov 02 14:24:48.807033 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.806965 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" event={"ID":"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4","Type":"ContainerDied","Data":"0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82"} Nov 02 14:24:48.807033 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.807021 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb" event={"ID":"74ccc3a9-7f72-467c-a8dd-b4783c6c08f4","Type":"ContainerDied","Data":"ac1d056caa24de8fa6931c7dd627fc301aa6e0a842b338a79179ba345f797de5"} Nov 02 14:24:48.807171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.807051 4431 scope.go:122] "RemoveContainer" containerID="0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82" Nov 02 14:24:48.807478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.807448 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:24:48.807535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.807523 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:24:48.807570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.807557 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:24:48.809594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.809570 4431 generic.go:356] "Generic (PLEG): container finished" podID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerID="e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13" exitCode=0 Nov 02 14:24:48.809704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.809679 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" Nov 02 14:24:48.809704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.809692 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" event={"ID":"3052f4d4-e528-4e62-8b00-b48e23ce005a","Type":"ContainerDied","Data":"e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13"} Nov 02 14:24:48.809846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.809733 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8" event={"ID":"3052f4d4-e528-4e62-8b00-b48e23ce005a","Type":"ContainerDied","Data":"f70bc6e677383162aab33900c2ea2cf6f8a8d89bd9e56c63a5d0e50ab5f0f417"} Nov 02 14:24:48.815387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.815363 4431 generic.go:356] "Generic (PLEG): container finished" podID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerID="fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243" exitCode=0 Nov 02 14:24:48.815502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.815432 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" Nov 02 14:24:48.815502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.815464 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" event={"ID":"2f32b447-e76e-42bc-a8fb-61a7ba9e9898","Type":"ContainerDied","Data":"fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243"} Nov 02 14:24:48.815502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.815498 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt" event={"ID":"2f32b447-e76e-42bc-a8fb-61a7ba9e9898","Type":"ContainerDied","Data":"1e6a92df939379866ef29fd980f5adae62a0c29bd463adfdf7bda067ff2f9f16"} Nov 02 14:24:48.826345 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.826325 4431 scope.go:122] "RemoveContainer" containerID="0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82" Nov 02 14:24:48.826644 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:24:48.826618 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82\": container with ID starting with 0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82 not found: ID does not exist" containerID="0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82" Nov 02 14:24:48.826693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.826654 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82"} err="failed to get container status \"0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82\": rpc error: code = NotFound desc = could not find container \"0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82\": container with ID starting with 0ca13ff1ebc9b1100ddf01886601c4671474dfdbdf2c5e9cb31de75d19289f82 not found: ID does not exist" Nov 02 14:24:48.826693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.826677 4431 scope.go:122] "RemoveContainer" containerID="e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13" Nov 02 14:24:48.837535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.837516 4431 scope.go:122] "RemoveContainer" containerID="e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13" Nov 02 14:24:48.837821 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:24:48.837803 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13\": container with ID starting with e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13 not found: ID does not exist" containerID="e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13" Nov 02 14:24:48.837892 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.837829 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13"} err="failed to get container status \"e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13\": rpc error: code = NotFound desc = could not find container \"e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13\": container with ID starting with e05a900dbb6b8795e86bf33f0b0d8565661258e49fe383ae6a2e6065e0f80d13 not found: ID does not exist" Nov 02 14:24:48.837892 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.837844 4431 scope.go:122] "RemoveContainer" containerID="fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243" Nov 02 14:24:48.848194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.848175 4431 scope.go:122] "RemoveContainer" containerID="fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243" Nov 02 14:24:48.848493 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:24:48.848473 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243\": container with ID starting with fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243 not found: ID does not exist" containerID="fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243" Nov 02 14:24:48.848566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.848501 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243"} err="failed to get container status \"fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243\": rpc error: code = NotFound desc = could not find container \"fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243\": container with ID starting with fbfa817846ea3ea1b1bf59a94f2bc0987a95b2b50025fad68b2ca9a197501243 not found: ID does not exist" Nov 02 14:24:48.895985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.895946 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt"] Nov 02 14:24:48.900231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.900205 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-klpjt"] Nov 02 14:24:48.938507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.938421 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb"] Nov 02 14:24:48.942940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.942901 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-p87xb"] Nov 02 14:24:48.980096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.980066 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8"] Nov 02 14:24:48.984928 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:48.984903 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-5fqt8"] Nov 02 14:24:49.034205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:49.034181 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974"] Nov 02 14:24:49.038098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:49.038078 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-711bc0df-z9974"] Nov 02 14:24:49.075465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:49.075437 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" path="/var/lib/kubelet/pods/2f32b447-e76e-42bc-a8fb-61a7ba9e9898/volumes" Nov 02 14:24:49.075934 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:49.075921 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" path="/var/lib/kubelet/pods/3052f4d4-e528-4e62-8b00-b48e23ce005a/volumes" Nov 02 14:24:49.076349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:49.076337 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" path="/var/lib/kubelet/pods/74ccc3a9-7f72-467c-a8dd-b4783c6c08f4/volumes" Nov 02 14:24:49.077058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:49.077044 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d4cc7b-5c3e-4334-a353-295faba67545" path="/var/lib/kubelet/pods/79d4cc7b-5c3e-4334-a353-295faba67545/volumes" Nov 02 14:24:49.827160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:49.827127 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" event={"ID":"80d74219-9bc6-492c-b1fb-01a2c2b22ebf","Type":"ContainerStarted","Data":"b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96"} Nov 02 14:24:50.517239 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:50.517212 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:24:50.517454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:50.517393 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:24:50.517538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:50.517457 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:24:51.304248 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:24:51.304192 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:24:51.516946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:51.516904 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:24:51.517160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:51.516963 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:24:52.517336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:52.517306 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:24:52.517802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:52.517357 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:24:53.517646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:53.517613 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:24:53.518108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:53.517665 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:24:54.517529 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:54.517497 4431 patch_prober.go:29] interesting pod/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 02 14:24:54.517723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:54.517547 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 02 14:24:55.521785 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:55.521731 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:24:55.523696 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:55.522130 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:24:55.523696 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:55.523224 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:24:58.656205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:58.656130 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:24:59.087844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.087744 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw"] Nov 02 14:24:59.088349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088331 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" Nov 02 14:24:59.088461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088350 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" Nov 02 14:24:59.088461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088363 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" Nov 02 14:24:59.088461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088368 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" Nov 02 14:24:59.088461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088396 4431 cpu_manager.go:414] "containerMap: removing container" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" Nov 02 14:24:59.088461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088401 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" Nov 02 14:24:59.088656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088484 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f32b447-e76e-42bc-a8fb-61a7ba9e9898" containerName="istio-proxy" Nov 02 14:24:59.088656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088494 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3052f4d4-e528-4e62-8b00-b48e23ce005a" containerName="istio-proxy" Nov 02 14:24:59.088656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.088504 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ccc3a9-7f72-467c-a8dd-b4783c6c08f4" containerName="istio-proxy" Nov 02 14:24:59.091524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.091505 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.110647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.110612 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-plugins-e2e-extmod-a-944506\"" Nov 02 14:24:59.110647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.110635 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-e06cd62782ae\"" Nov 02 14:24:59.111028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.111007 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"openshift-service-ca.crt\"" Nov 02 14:24:59.126128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.126100 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-e2e--4e96fc15-dockercfg-gffbp\"" Nov 02 14:24:59.126256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.126190 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"routing-overlay\"" Nov 02 14:24:59.126382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.126366 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"kube-root-ca.crt\"" Nov 02 14:24:59.128456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.128434 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw"] Nov 02 14:24:59.261248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.261217 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfmk\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-kube-api-access-jtfmk\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.261415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.261255 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.261415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.261279 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.261504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.261444 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.374841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.363132 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfmk\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-kube-api-access-jtfmk\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.374841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.363207 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.374841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.363250 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.374841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.363410 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.374841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.364234 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.374841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.364893 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.374841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.372212 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.392085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.392063 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfmk\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-kube-api-access-jtfmk\") pod \"payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.404969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.404941 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.716365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.716338 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw"] Nov 02 14:24:59.717551 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:24:59.717519 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490b2019_5e2f_475e_8e5c_9b9e38bb47cd.slice/crio-73536273df245e28b83299156195cd62f1f377e207c876ba410dcc7fdf8ab720 WatchSource:0}: Error finding container 73536273df245e28b83299156195cd62f1f377e207c876ba410dcc7fdf8ab720: Status 404 returned error can't find the container with id 73536273df245e28b83299156195cd62f1f377e207c876ba410dcc7fdf8ab720 Nov 02 14:24:59.919014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.917834 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" event={"ID":"490b2019-5e2f-475e-8e5c-9b9e38bb47cd","Type":"ContainerStarted","Data":"3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842"} Nov 02 14:24:59.919014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.917908 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" event={"ID":"490b2019-5e2f-475e-8e5c-9b9e38bb47cd","Type":"ContainerStarted","Data":"73536273df245e28b83299156195cd62f1f377e207c876ba410dcc7fdf8ab720"} Nov 02 14:24:59.919279 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.919020 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:24:59.966488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:24:59.966366 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" podStartSLOduration=0.966346484 podStartE2EDuration="966.346484ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:59.965040535 +0000 UTC m=+1665.002145467" watchObservedRunningTime="2026-11-02 14:24:59.966346484 +0000 UTC m=+1665.003451420" Nov 02 14:25:00.794799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.793017 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv"] Nov 02 14:25:00.820799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.802304 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv"] Nov 02 14:25:00.820799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.802418 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:00.831821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.831644 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"e2e-cfl-622143-provider-cred\"" Nov 02 14:25:00.902800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.895831 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:00.902800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.896037 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:00.902800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.896079 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:00.902800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.896158 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstmc\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-kube-api-access-cstmc\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.999477 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.999632 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.999671 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:00.999733 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstmc\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-kube-api-access-cstmc\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:01.001096 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:01.001642 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.001735 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.001753 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.002167 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.001887 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials podName:e838d955-be98-437d-a6b3-7e01d38ca2dd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:01.501846255 +0000 UTC m=+1666.538951182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.028808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:01.028749 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstmc\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-kube-api-access-cstmc\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.040446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:01.040418 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs"] Nov 02 14:25:01.044009 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:01.043979 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08d33ff-xwkbs"] Nov 02 14:25:01.081907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:01.081837 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8165b81-389a-4b7c-8d99-1f8a75f02988" path="/var/lib/kubelet/pods/e8165b81-389a-4b7c-8d99-1f8a75f02988/volumes" Nov 02 14:25:01.102187 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.102160 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.102367 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.102353 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.102518 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.102507 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials podName:490b2019-5e2f-475e-8e5c-9b9e38bb47cd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:01.602485946 +0000 UTC m=+1666.639590870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" (UID: "490b2019-5e2f-475e-8e5c-9b9e38bb47cd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.533346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:01.528257 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:01.533346 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.528671 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.533346 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.528699 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.533346 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.528934 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials podName:e838d955-be98-437d-a6b3-7e01d38ca2dd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:02.528902979 +0000 UTC m=+1667.566007911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.629744 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.629715 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.630022 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.630005 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:01.630268 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:01.630256 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials podName:490b2019-5e2f-475e-8e5c-9b9e38bb47cd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:02.630233039 +0000 UTC m=+1667.667337966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" (UID: "490b2019-5e2f-475e-8e5c-9b9e38bb47cd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:02.591748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.591705 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:02.592754 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:02.592430 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:02.592942 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:02.592929 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:02.593117 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:02.593105 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials podName:e838d955-be98-437d-a6b3-7e01d38ca2dd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:04.593083386 +0000 UTC m=+1669.630188313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:02.699611 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:02.698740 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:02.699611 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:02.698794 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:02.699611 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:02.698871 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials podName:490b2019-5e2f-475e-8e5c-9b9e38bb47cd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:04.69885071 +0000 UTC m=+1669.735955646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" (UID: "490b2019-5e2f-475e-8e5c-9b9e38bb47cd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:02.757181 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.757144 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw"] Nov 02 14:25:02.790195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.790158 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8"] Nov 02 14:25:02.797043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.797017 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:02.828133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.828101 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8"] Nov 02 14:25:02.912469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.912431 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46hw\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-kube-api-access-g46hw\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:02.913431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.913403 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:02.913680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.913662 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:02.913966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:02.913945 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.029945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.029907 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46hw\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-kube-api-access-g46hw\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.030210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.030189 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.030370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.030350 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.030600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.030579 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.031748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.031725 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.034419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.034395 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.103491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.103374 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.104558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.104505 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46hw\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-kube-api-access-g46hw\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.186857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.184685 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:03.756928 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:03.756903 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8"] Nov 02 14:25:03.758261 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:03.758229 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93655790_866b_4892_a19c_129b9365f3d0.slice/crio-5eaa138241168a283cab0c997592a4a41fc51a3be1e533f62af65dbe87975583 WatchSource:0}: Error finding container 5eaa138241168a283cab0c997592a4a41fc51a3be1e533f62af65dbe87975583: Status 404 returned error can't find the container with id 5eaa138241168a283cab0c997592a4a41fc51a3be1e533f62af65dbe87975583 Nov 02 14:25:04.023425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:04.023334 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" event={"ID":"93655790-866b-4892-a19c-129b9365f3d0","Type":"ContainerStarted","Data":"28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283"} Nov 02 14:25:04.023425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:04.023402 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:04.023611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:04.023426 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" event={"ID":"93655790-866b-4892-a19c-129b9365f3d0","Type":"ContainerStarted","Data":"5eaa138241168a283cab0c997592a4a41fc51a3be1e533f62af65dbe87975583"} Nov 02 14:25:04.073466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:04.073407 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" podStartSLOduration=2.073388213 podStartE2EDuration="2.073388213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:04.071554905 +0000 UTC m=+1669.108659837" watchObservedRunningTime="2026-11-02 14:25:04.073388213 +0000 UTC m=+1669.110493148" Nov 02 14:25:04.676922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:04.676886 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:04.677105 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:04.677050 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:04.677105 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:04.677073 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:04.677180 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:04.677140 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials podName:e838d955-be98-437d-a6b3-7e01d38ca2dd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:08.677123345 +0000 UTC m=+1673.714228275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:04.778498 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:04.778466 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:04.778498 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:04.778494 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:04.778961 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:04.778553 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials podName:490b2019-5e2f-475e-8e5c-9b9e38bb47cd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:08.778537194 +0000 UTC m=+1673.815642117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" (UID: "490b2019-5e2f-475e-8e5c-9b9e38bb47cd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:08.015367 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.015269 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" podUID="490b2019-5e2f-475e-8e5c-9b9e38bb47cd" containerName="payload-processing" containerID="cri-o://3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842" gracePeriod=25 Nov 02 14:25:08.284948 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.284927 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:25:08.318240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.318213 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-kube-api-access-jtfmk\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-kube-api-access-jtfmk\") pod \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " Nov 02 14:25:08.318344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.318257 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials\") pod \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " Nov 02 14:25:08.318344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.318300 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-plugins-config-volume\") pod \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " Nov 02 14:25:08.318344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.318321 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-routing-overlay-volume\") pod \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\" (UID: \"490b2019-5e2f-475e-8e5c-9b9e38bb47cd\") " Nov 02 14:25:08.318705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.318680 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-plugins-config-volume" pod "490b2019-5e2f-475e-8e5c-9b9e38bb47cd" (UID: "490b2019-5e2f-475e-8e5c-9b9e38bb47cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:08.318843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.318692 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-routing-overlay-volume" pod "490b2019-5e2f-475e-8e5c-9b9e38bb47cd" (UID: "490b2019-5e2f-475e-8e5c-9b9e38bb47cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:08.320456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.320431 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials" pod "490b2019-5e2f-475e-8e5c-9b9e38bb47cd" (UID: "490b2019-5e2f-475e-8e5c-9b9e38bb47cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:08.320551 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.320480 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-kube-api-access-jtfmk" pod "490b2019-5e2f-475e-8e5c-9b9e38bb47cd" (UID: "490b2019-5e2f-475e-8e5c-9b9e38bb47cd"). InnerVolumeSpecName "kube-api-access-jtfmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:08.420219 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.420190 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtfmk\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-kube-api-access-jtfmk\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:08.420219 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.420218 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:08.420394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.420229 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:08.420394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.420238 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/490b2019-5e2f-475e-8e5c-9b9e38bb47cd-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:08.724130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.724095 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:08.724324 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:08.724258 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:08.724324 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:08.724284 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:08.724460 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:08.724352 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials podName:e838d955-be98-437d-a6b3-7e01d38ca2dd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:16.724334022 +0000 UTC m=+1681.761438952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:08.869955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.869925 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc"] Nov 02 14:25:08.870524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.870511 4431 cpu_manager.go:414] "containerMap: removing container" podUID="490b2019-5e2f-475e-8e5c-9b9e38bb47cd" containerName="payload-processing" Nov 02 14:25:08.870577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.870526 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490b2019-5e2f-475e-8e5c-9b9e38bb47cd" containerName="payload-processing" Nov 02 14:25:08.870628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.870618 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="490b2019-5e2f-475e-8e5c-9b9e38bb47cd" containerName="payload-processing" Nov 02 14:25:08.874123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.874108 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:08.892944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.892922 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:25:08.893120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.893027 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:25:08.896132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.896109 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc"] Nov 02 14:25:08.927271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.927246 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:08.927382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.927279 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:08.927424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:08.927406 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6bhb\" (UniqueName: \"kubernetes.io/projected/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-kube-api-access-g6bhb\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:09.029128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.029052 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:09.029128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.029090 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:09.029539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.029157 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bhb\" (UniqueName: \"kubernetes.io/projected/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-kube-api-access-g6bhb\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:09.029859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.029837 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:09.031395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.031375 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:09.054155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.054130 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bhb\" (UniqueName: \"kubernetes.io/projected/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-kube-api-access-g6bhb\") pod \"maas-api-revoke-sub-6fe456d6-ncmcc\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:09.058088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.058055 4431 generic.go:356] "Generic (PLEG): container finished" podID="490b2019-5e2f-475e-8e5c-9b9e38bb47cd" containerID="3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842" exitCode=0 Nov 02 14:25:09.058195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.058097 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" Nov 02 14:25:09.058195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.058128 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" event={"ID":"490b2019-5e2f-475e-8e5c-9b9e38bb47cd","Type":"ContainerDied","Data":"3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842"} Nov 02 14:25:09.058195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.058178 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw" event={"ID":"490b2019-5e2f-475e-8e5c-9b9e38bb47cd","Type":"ContainerDied","Data":"73536273df245e28b83299156195cd62f1f377e207c876ba410dcc7fdf8ab720"} Nov 02 14:25:09.058319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.058200 4431 scope.go:122] "RemoveContainer" containerID="3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842" Nov 02 14:25:09.070265 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.070245 4431 scope.go:122] "RemoveContainer" containerID="3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842" Nov 02 14:25:09.070542 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:09.070522 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842\": container with ID starting with 3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842 not found: ID does not exist" containerID="3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842" Nov 02 14:25:09.070606 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.070553 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842"} err="failed to get container status \"3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842\": rpc error: code = NotFound desc = could not find container \"3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842\": container with ID starting with 3bdc4a82c5360b0dcda8570e4e18eb1eed3269d6a209092f5df7fe39d15b4842 not found: ID does not exist" Nov 02 14:25:09.115241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.115216 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw"] Nov 02 14:25:09.119134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.119114 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-7c6fvlmvw"] Nov 02 14:25:09.187796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.187754 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:09.494936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:09.494912 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc"] Nov 02 14:25:09.496430 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:09.496404 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cbd4eb_71ef_4fd8_83e3_7538c2606205.slice/crio-7e6188b5fb243dcc7e5d53f35b074d69247769e1eea18a57a65562d33ec398aa WatchSource:0}: Error finding container 7e6188b5fb243dcc7e5d53f35b074d69247769e1eea18a57a65562d33ec398aa: Status 404 returned error can't find the container with id 7e6188b5fb243dcc7e5d53f35b074d69247769e1eea18a57a65562d33ec398aa Nov 02 14:25:10.066890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:10.066812 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" event={"ID":"e4cbd4eb-71ef-4fd8-83e3-7538c2606205","Type":"ContainerStarted","Data":"b1e3652e6832bde217ff33e838a4cb3972a0576e8703a1e7e5b80af37e1325f6"} Nov 02 14:25:10.067246 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:10.066895 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" event={"ID":"e4cbd4eb-71ef-4fd8-83e3-7538c2606205","Type":"ContainerStarted","Data":"7e6188b5fb243dcc7e5d53f35b074d69247769e1eea18a57a65562d33ec398aa"} Nov 02 14:25:10.112035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:10.111988 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" podStartSLOduration=2.1119750330000002 podStartE2EDuration="2.111975033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:10.109144954 +0000 UTC m=+1675.146249885" watchObservedRunningTime="2026-11-02 14:25:10.111975033 +0000 UTC m=+1675.149080016" Nov 02 14:25:11.074719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:11.074674 4431 generic.go:356] "Generic (PLEG): container finished" podID="e4cbd4eb-71ef-4fd8-83e3-7538c2606205" containerID="b1e3652e6832bde217ff33e838a4cb3972a0576e8703a1e7e5b80af37e1325f6" exitCode=0 Nov 02 14:25:11.076427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:11.076398 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490b2019-5e2f-475e-8e5c-9b9e38bb47cd" path="/var/lib/kubelet/pods/490b2019-5e2f-475e-8e5c-9b9e38bb47cd/volumes" Nov 02 14:25:11.076755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:11.076734 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" event={"ID":"e4cbd4eb-71ef-4fd8-83e3-7538c2606205","Type":"ContainerDied","Data":"b1e3652e6832bde217ff33e838a4cb3972a0576e8703a1e7e5b80af37e1325f6"} Nov 02 14:25:12.230111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.230087 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:12.260407 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.260382 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-xks-ca\") pod \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " Nov 02 14:25:12.260522 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.260475 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-openshift-ca\") pod \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " Nov 02 14:25:12.260686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.260672 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-kube-api-access-g6bhb\" (UniqueName: \"kubernetes.io/projected/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-kube-api-access-g6bhb\") pod \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\" (UID: \"e4cbd4eb-71ef-4fd8-83e3-7538c2606205\") " Nov 02 14:25:12.260844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.260820 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-openshift-ca" pod "e4cbd4eb-71ef-4fd8-83e3-7538c2606205" (UID: "e4cbd4eb-71ef-4fd8-83e3-7538c2606205"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:12.261154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.261135 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:12.262454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.262434 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-xks-ca" pod "e4cbd4eb-71ef-4fd8-83e3-7538c2606205" (UID: "e4cbd4eb-71ef-4fd8-83e3-7538c2606205"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:25:12.262685 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.262669 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-kube-api-access-g6bhb" pod "e4cbd4eb-71ef-4fd8-83e3-7538c2606205" (UID: "e4cbd4eb-71ef-4fd8-83e3-7538c2606205"). InnerVolumeSpecName "kube-api-access-g6bhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:12.362577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.362515 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6bhb\" (UniqueName: \"kubernetes.io/projected/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-kube-api-access-g6bhb\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:12.362577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:12.362539 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cbd4eb-71ef-4fd8-83e3-7538c2606205-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:13.089464 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:13.089442 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" Nov 02 14:25:13.089598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:13.089435 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc" event={"ID":"e4cbd4eb-71ef-4fd8-83e3-7538c2606205","Type":"ContainerDied","Data":"7e6188b5fb243dcc7e5d53f35b074d69247769e1eea18a57a65562d33ec398aa"} Nov 02 14:25:13.089598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:13.089568 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e6188b5fb243dcc7e5d53f35b074d69247769e1eea18a57a65562d33ec398aa" Nov 02 14:25:15.032930 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:15.032905 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:25:15.122595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:15.122566 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv"] Nov 02 14:25:15.123020 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:15.123000 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" podUID="e838d955-be98-437d-a6b3-7e01d38ca2dd" Nov 02 14:25:16.110125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.110087 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:16.117392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.117366 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:16.210735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.210705 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-plugins-config-volume\") pod \"e838d955-be98-437d-a6b3-7e01d38ca2dd\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " Nov 02 14:25:16.210950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.210757 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-routing-overlay-volume\") pod \"e838d955-be98-437d-a6b3-7e01d38ca2dd\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " Nov 02 14:25:16.210950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.210830 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-kube-api-access-cstmc\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-kube-api-access-cstmc\") pod \"e838d955-be98-437d-a6b3-7e01d38ca2dd\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " Nov 02 14:25:16.211157 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.211132 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-routing-overlay-volume" pod "e838d955-be98-437d-a6b3-7e01d38ca2dd" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:16.211233 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.211137 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-plugins-config-volume" pod "e838d955-be98-437d-a6b3-7e01d38ca2dd" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:16.212994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.212971 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-kube-api-access-cstmc" pod "e838d955-be98-437d-a6b3-7e01d38ca2dd" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd"). InnerVolumeSpecName "kube-api-access-cstmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:16.312597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.312576 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:16.312597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.312598 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e838d955-be98-437d-a6b3-7e01d38ca2dd-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:16.312721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.312608 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cstmc\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-kube-api-access-cstmc\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:16.817192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.817155 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv\" (UID: \"e838d955-be98-437d-a6b3-7e01d38ca2dd\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:16.817381 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:16.817293 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-e06cd62782ae: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:16.817381 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:16.817312 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv: secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:16.817381 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:16.817365 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials podName:e838d955-be98-437d-a6b3-7e01d38ca2dd nodeName:}" failed. No retries permitted until 2026-11-02 14:25:32.817349454 +0000 UTC m=+1697.854454385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" (UID: "e838d955-be98-437d-a6b3-7e01d38ca2dd") : secret "agc-cred-e06cd62782ae" not found Nov 02 14:25:16.946562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.946529 4431 scope.go:122] "RemoveContainer" containerID="555e7278dcdb92002a158f8f76c63ff6e93a16b974643486b142cfcff1afaac7" Nov 02 14:25:16.958483 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.958460 4431 scope.go:122] "RemoveContainer" containerID="5250e397fb962b329817380b95f14d4fbbe63884c6ed669358afb7a04243e340" Nov 02 14:25:16.969201 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.969176 4431 scope.go:122] "RemoveContainer" containerID="bc65c4c8b5056e10159bc322ea3cada2d3952f48c40413483ec20e866ad2e313" Nov 02 14:25:16.978574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.978551 4431 scope.go:122] "RemoveContainer" containerID="edf7f13389113ed83d0a43574cc0dc3bbc5c5505ae5ecca223eb32b130d19371" Nov 02 14:25:16.988025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:16.987998 4431 scope.go:122] "RemoveContainer" containerID="e1ef917a53d451a6b822a6bcfb24aa52db08dead5f029d29d2fbf660298f3edd" Nov 02 14:25:17.116352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:17.116325 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv" Nov 02 14:25:17.204093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:17.204060 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv"] Nov 02 14:25:17.210921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:17.210895 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-79c5hq2sv"] Nov 02 14:25:17.322126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:17.322096 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e838d955-be98-437d-a6b3-7e01d38ca2dd-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:19.076575 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:19.076541 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e838d955-be98-437d-a6b3-7e01d38ca2dd" path="/var/lib/kubelet/pods/e838d955-be98-437d-a6b3-7e01d38ca2dd/volumes" Nov 02 14:25:21.282848 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:21.282810 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:25:21.433828 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.433791 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7"] Nov 02 14:25:21.434428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.434409 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e4cbd4eb-71ef-4fd8-83e3-7538c2606205" containerName="revoke-keys" Nov 02 14:25:21.434539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.434429 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cbd4eb-71ef-4fd8-83e3-7538c2606205" containerName="revoke-keys" Nov 02 14:25:21.434594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.434567 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cbd4eb-71ef-4fd8-83e3-7538c2606205" containerName="revoke-keys" Nov 02 14:25:21.439860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.439837 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.464677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.464635 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7"] Nov 02 14:25:21.514762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.514737 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:25:21.514932 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.514872 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:25:21.571661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.571594 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.571661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.571642 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.571861 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.571838 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs25k\" (UniqueName: \"kubernetes.io/projected/5ca69dbf-37f9-4a28-944b-5da842f20991-kube-api-access-bs25k\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.672770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.672738 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.672906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.672807 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.672906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.672888 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs25k\" (UniqueName: \"kubernetes.io/projected/5ca69dbf-37f9-4a28-944b-5da842f20991-kube-api-access-bs25k\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.673519 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.673497 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.675159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.675131 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.707113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.707085 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs25k\" (UniqueName: \"kubernetes.io/projected/5ca69dbf-37f9-4a28-944b-5da842f20991-kube-api-access-bs25k\") pod \"maas-api-revoke-sub-ad18a670-zn6w7\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:21.752817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:21.752798 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:22.054260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:22.054234 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7"] Nov 02 14:25:22.055389 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:22.055363 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca69dbf_37f9_4a28_944b_5da842f20991.slice/crio-9d6eb6b18998046dcbde200fc0fef20cc4ade5d159475d44b479442095104bf5 WatchSource:0}: Error finding container 9d6eb6b18998046dcbde200fc0fef20cc4ade5d159475d44b479442095104bf5: Status 404 returned error can't find the container with id 9d6eb6b18998046dcbde200fc0fef20cc4ade5d159475d44b479442095104bf5 Nov 02 14:25:22.158124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:22.158098 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" event={"ID":"5ca69dbf-37f9-4a28-944b-5da842f20991","Type":"ContainerStarted","Data":"9d6eb6b18998046dcbde200fc0fef20cc4ade5d159475d44b479442095104bf5"} Nov 02 14:25:23.165754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:23.165676 4431 generic.go:356] "Generic (PLEG): container finished" podID="5ca69dbf-37f9-4a28-944b-5da842f20991" containerID="48e4cb0bbc02f6d2665e06072c81b363f81317d9d899d0830e7a6eb19dad5914" exitCode=0 Nov 02 14:25:23.165754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:23.165744 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" event={"ID":"5ca69dbf-37f9-4a28-944b-5da842f20991","Type":"ContainerDied","Data":"48e4cb0bbc02f6d2665e06072c81b363f81317d9d899d0830e7a6eb19dad5914"} Nov 02 14:25:24.317498 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.317474 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:24.402598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.402571 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-openshift-ca\") pod \"5ca69dbf-37f9-4a28-944b-5da842f20991\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " Nov 02 14:25:24.402766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.402630 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca69dbf-37f9-4a28-944b-5da842f20991-kube-api-access-bs25k\" (UniqueName: \"kubernetes.io/projected/5ca69dbf-37f9-4a28-944b-5da842f20991-kube-api-access-bs25k\") pod \"5ca69dbf-37f9-4a28-944b-5da842f20991\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " Nov 02 14:25:24.402766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.402664 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-xks-ca\") pod \"5ca69dbf-37f9-4a28-944b-5da842f20991\" (UID: \"5ca69dbf-37f9-4a28-944b-5da842f20991\") " Nov 02 14:25:24.403024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.402998 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-openshift-ca" pod "5ca69dbf-37f9-4a28-944b-5da842f20991" (UID: "5ca69dbf-37f9-4a28-944b-5da842f20991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:24.403307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.403286 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:24.404703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.404678 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-xks-ca" pod "5ca69dbf-37f9-4a28-944b-5da842f20991" (UID: "5ca69dbf-37f9-4a28-944b-5da842f20991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:25:24.404811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.404765 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca69dbf-37f9-4a28-944b-5da842f20991-kube-api-access-bs25k" pod "5ca69dbf-37f9-4a28-944b-5da842f20991" (UID: "5ca69dbf-37f9-4a28-944b-5da842f20991"). InnerVolumeSpecName "kube-api-access-bs25k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:24.504802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.504715 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs25k\" (UniqueName: \"kubernetes.io/projected/5ca69dbf-37f9-4a28-944b-5da842f20991-kube-api-access-bs25k\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:24.504802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:24.504743 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca69dbf-37f9-4a28-944b-5da842f20991-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:25.181470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:25.181443 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" Nov 02 14:25:25.181632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:25.181476 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7" event={"ID":"5ca69dbf-37f9-4a28-944b-5da842f20991","Type":"ContainerDied","Data":"9d6eb6b18998046dcbde200fc0fef20cc4ade5d159475d44b479442095104bf5"} Nov 02 14:25:25.181632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:25.181546 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6eb6b18998046dcbde200fc0fef20cc4ade5d159475d44b479442095104bf5" Nov 02 14:25:27.467451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:27.467420 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:25:31.101811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.095875 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:25:31.101811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.095951 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:25:31.127893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.127858 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:25:31.128150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.128121 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:25:31.180894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.180851 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:25:31.288496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.288454 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:25:31.377859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.375529 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:25:31.428913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.428869 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:25:31.491370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.491327 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:25:31.524891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.524848 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 14:25:31.766801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.757583 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td"] Nov 02 14:25:31.766801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.758517 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5ca69dbf-37f9-4a28-944b-5da842f20991" containerName="revoke-keys" Nov 02 14:25:31.766801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.758540 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca69dbf-37f9-4a28-944b-5da842f20991" containerName="revoke-keys" Nov 02 14:25:31.766801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.758706 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca69dbf-37f9-4a28-944b-5da842f20991" containerName="revoke-keys" Nov 02 14:25:31.794982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.794956 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td"] Nov 02 14:25:31.795329 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.795270 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:31.820117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.820091 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:25:31.820943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.820923 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:25:31.940810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.925398 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:31.940810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.925454 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:31.940810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:31.925544 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dw4j\" (UniqueName: \"kubernetes.io/projected/e549b7f5-6024-4912-b654-fca247d92fbe-kube-api-access-7dw4j\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:32.066847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:32.028530 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:32.066847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:32.028588 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:32.066847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:32.028700 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dw4j\" (UniqueName: \"kubernetes.io/projected/e549b7f5-6024-4912-b654-fca247d92fbe-kube-api-access-7dw4j\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:32.066847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:32.030306 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:32.066847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:32.054605 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:32.099661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:32.099553 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dw4j\" (UniqueName: \"kubernetes.io/projected/e549b7f5-6024-4912-b654-fca247d92fbe-kube-api-access-7dw4j\") pod \"maas-api-revoke-keys-e2e-ait-b3690233-md7td\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:32.224527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:32.224494 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:32.565543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:32.565515 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td"] Nov 02 14:25:32.566740 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:32.566712 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode549b7f5_6024_4912_b654_fca247d92fbe.slice/crio-4b64619e3225a4fa2d65828f814e9b3eaeee6473ebb279cffe14db3306b7c9ce WatchSource:0}: Error finding container 4b64619e3225a4fa2d65828f814e9b3eaeee6473ebb279cffe14db3306b7c9ce: Status 404 returned error can't find the container with id 4b64619e3225a4fa2d65828f814e9b3eaeee6473ebb279cffe14db3306b7c9ce Nov 02 14:25:33.316472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:33.316381 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" event={"ID":"e549b7f5-6024-4912-b654-fca247d92fbe","Type":"ContainerStarted","Data":"06c5ccebf6e6686fa681e6119ed7a5da95e825778a9172240b90c2e67429625b"} Nov 02 14:25:33.316472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:33.316439 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" event={"ID":"e549b7f5-6024-4912-b654-fca247d92fbe","Type":"ContainerStarted","Data":"4b64619e3225a4fa2d65828f814e9b3eaeee6473ebb279cffe14db3306b7c9ce"} Nov 02 14:25:33.360356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:33.360303 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" podStartSLOduration=2.360289105 podStartE2EDuration="2.360289105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:33.360095473 +0000 UTC m=+1698.397200407" watchObservedRunningTime="2026-11-02 14:25:33.360289105 +0000 UTC m=+1698.397394039" Nov 02 14:25:34.166286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.166248 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-469647"] Nov 02 14:25:34.170602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.170577 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:34.186847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.186821 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-469647"] Nov 02 14:25:34.196034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.196009 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nz2z6\"" Nov 02 14:25:34.264999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.264967 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxkz\" (UniqueName: \"kubernetes.io/projected/91d197e7-8539-4735-8df0-2deb84e20cf9-kube-api-access-zdxkz\") pod \"test-curl-7816-469647\" (UID: \"91d197e7-8539-4735-8df0-2deb84e20cf9\") " pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:34.324361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.324331 4431 generic.go:356] "Generic (PLEG): container finished" podID="e549b7f5-6024-4912-b654-fca247d92fbe" containerID="06c5ccebf6e6686fa681e6119ed7a5da95e825778a9172240b90c2e67429625b" exitCode=0 Nov 02 14:25:34.324762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.324414 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" event={"ID":"e549b7f5-6024-4912-b654-fca247d92fbe","Type":"ContainerDied","Data":"06c5ccebf6e6686fa681e6119ed7a5da95e825778a9172240b90c2e67429625b"} Nov 02 14:25:34.365828 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.365763 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxkz\" (UniqueName: \"kubernetes.io/projected/91d197e7-8539-4735-8df0-2deb84e20cf9-kube-api-access-zdxkz\") pod \"test-curl-7816-469647\" (UID: \"91d197e7-8539-4735-8df0-2deb84e20cf9\") " pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:34.391385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.391357 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxkz\" (UniqueName: \"kubernetes.io/projected/91d197e7-8539-4735-8df0-2deb84e20cf9-kube-api-access-zdxkz\") pod \"test-curl-7816-469647\" (UID: \"91d197e7-8539-4735-8df0-2deb84e20cf9\") " pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:34.485555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.485482 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:34.793263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:34.793228 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-469647"] Nov 02 14:25:34.794266 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:34.794239 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91d197e7_8539_4735_8df0_2deb84e20cf9.slice/crio-07f01959bb330ae35ba4a768ad7e46f232ff935da7290d1eddd2255ad914e457 WatchSource:0}: Error finding container 07f01959bb330ae35ba4a768ad7e46f232ff935da7290d1eddd2255ad914e457: Status 404 returned error can't find the container with id 07f01959bb330ae35ba4a768ad7e46f232ff935da7290d1eddd2255ad914e457 Nov 02 14:25:35.334741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.334703 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-469647" event={"ID":"91d197e7-8539-4735-8df0-2deb84e20cf9","Type":"ContainerStarted","Data":"0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e"} Nov 02 14:25:35.335224 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.334753 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-469647" event={"ID":"91d197e7-8539-4735-8df0-2deb84e20cf9","Type":"ContainerStarted","Data":"07f01959bb330ae35ba4a768ad7e46f232ff935da7290d1eddd2255ad914e457"} Nov 02 14:25:35.376555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.376502 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7816-469647" podStartSLOduration=1.376488561 podStartE2EDuration="1.376488561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:35.375106967 +0000 UTC m=+1700.412211902" watchObservedRunningTime="2026-11-02 14:25:35.376488561 +0000 UTC m=+1700.413593493" Nov 02 14:25:35.505859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.505835 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:35.581449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.581308 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-xks-ca\") pod \"e549b7f5-6024-4912-b654-fca247d92fbe\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " Nov 02 14:25:35.581629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.581570 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-openshift-ca\") pod \"e549b7f5-6024-4912-b654-fca247d92fbe\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " Nov 02 14:25:35.582098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.582069 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-openshift-ca" pod "e549b7f5-6024-4912-b654-fca247d92fbe" (UID: "e549b7f5-6024-4912-b654-fca247d92fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:25:35.582541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.582516 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e549b7f5-6024-4912-b654-fca247d92fbe-kube-api-access-7dw4j\" (UniqueName: \"kubernetes.io/projected/e549b7f5-6024-4912-b654-fca247d92fbe-kube-api-access-7dw4j\") pod \"e549b7f5-6024-4912-b654-fca247d92fbe\" (UID: \"e549b7f5-6024-4912-b654-fca247d92fbe\") " Nov 02 14:25:35.583087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.583063 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:35.585068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.583932 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-xks-ca" pod "e549b7f5-6024-4912-b654-fca247d92fbe" (UID: "e549b7f5-6024-4912-b654-fca247d92fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:25:35.585625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.585600 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e549b7f5-6024-4912-b654-fca247d92fbe-kube-api-access-7dw4j" pod "e549b7f5-6024-4912-b654-fca247d92fbe" (UID: "e549b7f5-6024-4912-b654-fca247d92fbe"). InnerVolumeSpecName "kube-api-access-7dw4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:35.683913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.683886 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dw4j\" (UniqueName: \"kubernetes.io/projected/e549b7f5-6024-4912-b654-fca247d92fbe-kube-api-access-7dw4j\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:35.683913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.683911 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e549b7f5-6024-4912-b654-fca247d92fbe-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:35.732956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.732913 4431 patch_prober.go:29] interesting pod/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:25:35.733101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:35.732985 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:25:36.245212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.245183 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-469647"] Nov 02 14:25:36.248440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.248413 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-469647"] Nov 02 14:25:36.342707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.342677 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" Nov 02 14:25:36.343174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.342699 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td" event={"ID":"e549b7f5-6024-4912-b654-fca247d92fbe","Type":"ContainerDied","Data":"4b64619e3225a4fa2d65828f814e9b3eaeee6473ebb279cffe14db3306b7c9ce"} Nov 02 14:25:36.343174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.342754 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b64619e3225a4fa2d65828f814e9b3eaeee6473ebb279cffe14db3306b7c9ce" Nov 02 14:25:36.402861 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.402833 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-f108c6"] Nov 02 14:25:36.403369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.403354 4431 cpu_manager.go:414] "containerMap: removing container" podUID="91d197e7-8539-4735-8df0-2deb84e20cf9" containerName="test-curl-7816-469647" Nov 02 14:25:36.403429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.403370 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d197e7-8539-4735-8df0-2deb84e20cf9" containerName="test-curl-7816-469647" Nov 02 14:25:36.403429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.403396 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e549b7f5-6024-4912-b654-fca247d92fbe" containerName="revoke-keys" Nov 02 14:25:36.403429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.403402 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e549b7f5-6024-4912-b654-fca247d92fbe" containerName="revoke-keys" Nov 02 14:25:36.407820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.404185 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d197e7-8539-4735-8df0-2deb84e20cf9" containerName="test-curl-7816-469647" Nov 02 14:25:36.407820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.404231 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e549b7f5-6024-4912-b654-fca247d92fbe" containerName="revoke-keys" Nov 02 14:25:36.410095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.410069 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-f108c6" Nov 02 14:25:36.430680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.428451 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-f108c6"] Nov 02 14:25:36.492634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.492595 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzs92\" (UniqueName: \"kubernetes.io/projected/8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75-kube-api-access-dzs92\") pod \"test-curl-7816-f108c6\" (UID: \"8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75\") " pod="opendatahub/test-curl-7816-f108c6" Nov 02 14:25:36.593891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.593813 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzs92\" (UniqueName: \"kubernetes.io/projected/8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75-kube-api-access-dzs92\") pod \"test-curl-7816-f108c6\" (UID: \"8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75\") " pod="opendatahub/test-curl-7816-f108c6" Nov 02 14:25:36.618737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.618708 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzs92\" (UniqueName: \"kubernetes.io/projected/8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75-kube-api-access-dzs92\") pod \"test-curl-7816-f108c6\" (UID: \"8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75\") " pod="opendatahub/test-curl-7816-f108c6" Nov 02 14:25:36.734941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:36.734908 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-f108c6" Nov 02 14:25:37.039225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.039193 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-f108c6"] Nov 02 14:25:37.040219 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:37.040192 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b5f4a0f_9ba0_4882_a5ee_6bf1e26c2a75.slice/crio-b8d1cbd821b4429a0456829170d84103525dea310972c73c1fc935f43c9343bd WatchSource:0}: Error finding container b8d1cbd821b4429a0456829170d84103525dea310972c73c1fc935f43c9343bd: Status 404 returned error can't find the container with id b8d1cbd821b4429a0456829170d84103525dea310972c73c1fc935f43c9343bd Nov 02 14:25:37.351456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.351427 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-f108c6" event={"ID":"8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75","Type":"ContainerStarted","Data":"6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d"} Nov 02 14:25:37.351913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.351477 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-f108c6" event={"ID":"8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75","Type":"ContainerStarted","Data":"b8d1cbd821b4429a0456829170d84103525dea310972c73c1fc935f43c9343bd"} Nov 02 14:25:37.353488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.353461 4431 generic.go:356] "Generic (PLEG): container finished" podID="181db062-305e-4f1f-942c-3b1feed06efe" containerID="fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2" exitCode=0 Nov 02 14:25:37.353613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.353535 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" event={"ID":"181db062-305e-4f1f-942c-3b1feed06efe","Type":"ContainerDied","Data":"fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2"} Nov 02 14:25:37.353613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.353574 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7816-469647" podUID="91d197e7-8539-4735-8df0-2deb84e20cf9" containerName="test-curl-7816-469647" containerID="cri-o://0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e" gracePeriod=2 Nov 02 14:25:37.353975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.353959 4431 scope.go:122] "RemoveContainer" containerID="fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2" Nov 02 14:25:37.355433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.355398 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:37.355526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.355496 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:37.355575 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.355561 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:37.397616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:37.397576 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7816-f108c6" podStartSLOduration=1.397563974 podStartE2EDuration="1.397563974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:37.394608621 +0000 UTC m=+1702.431713556" watchObservedRunningTime="2026-11-02 14:25:37.397563974 +0000 UTC m=+1702.434668905" Nov 02 14:25:38.349228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.349191 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-f108c6"] Nov 02 14:25:38.359726 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.357329 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-f108c6"] Nov 02 14:25:38.365101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.365072 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" event={"ID":"181db062-305e-4f1f-942c-3b1feed06efe","Type":"ContainerStarted","Data":"4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05"} Nov 02 14:25:38.513347 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.513315 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-4986cc"] Nov 02 14:25:38.514131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.514110 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75" containerName="test-curl-7816-f108c6" Nov 02 14:25:38.514131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.514131 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75" containerName="test-curl-7816-f108c6" Nov 02 14:25:38.514304 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.514258 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75" containerName="test-curl-7816-f108c6" Nov 02 14:25:38.516669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.516643 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:38.535696 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.535676 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-4986cc"] Nov 02 14:25:38.625299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.625202 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbq6z\" (UniqueName: \"kubernetes.io/projected/c4930f88-1289-4622-b085-6560b05affc9-kube-api-access-hbq6z\") pod \"test-curl-7816-4986cc\" (UID: \"c4930f88-1289-4622-b085-6560b05affc9\") " pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:38.726581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.726548 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbq6z\" (UniqueName: \"kubernetes.io/projected/c4930f88-1289-4622-b085-6560b05affc9-kube-api-access-hbq6z\") pod \"test-curl-7816-4986cc\" (UID: \"c4930f88-1289-4622-b085-6560b05affc9\") " pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:38.751460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.751425 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbq6z\" (UniqueName: \"kubernetes.io/projected/c4930f88-1289-4622-b085-6560b05affc9-kube-api-access-hbq6z\") pod \"test-curl-7816-4986cc\" (UID: \"c4930f88-1289-4622-b085-6560b05affc9\") " pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:38.827854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.827827 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:38.836728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.836706 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:25:38.836925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.836903 4431 patch_prober.go:29] interesting pod/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:25:38.837014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:38.836987 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:25:39.160750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.160725 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-4986cc"] Nov 02 14:25:39.161328 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:39.161296 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4930f88_1289_4622_b085_6560b05affc9.slice/crio-ea913ee108f55ee11c73222d56a90e89187e90f4c6e466a6eb88b183d3420376 WatchSource:0}: Error finding container ea913ee108f55ee11c73222d56a90e89187e90f4c6e466a6eb88b183d3420376: Status 404 returned error can't find the container with id ea913ee108f55ee11c73222d56a90e89187e90f4c6e466a6eb88b183d3420376 Nov 02 14:25:39.397816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.397712 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-4986cc" event={"ID":"c4930f88-1289-4622-b085-6560b05affc9","Type":"ContainerStarted","Data":"f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9"} Nov 02 14:25:39.397816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.397765 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-4986cc" event={"ID":"c4930f88-1289-4622-b085-6560b05affc9","Type":"ContainerStarted","Data":"ea913ee108f55ee11c73222d56a90e89187e90f4c6e466a6eb88b183d3420376"} Nov 02 14:25:39.398235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.398127 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7816-f108c6" podUID="8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75" containerName="test-curl-7816-f108c6" containerID="cri-o://6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d" gracePeriod=2 Nov 02 14:25:39.438929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.438835 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7816-4986cc" podStartSLOduration=1.438817083 podStartE2EDuration="1.438817083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:39.438387803 +0000 UTC m=+1704.475492735" watchObservedRunningTime="2026-11-02 14:25:39.438817083 +0000 UTC m=+1704.475922014" Nov 02 14:25:39.564221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.564198 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:39.637507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.637475 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d197e7-8539-4735-8df0-2deb84e20cf9-kube-api-access-zdxkz\" (UniqueName: \"kubernetes.io/projected/91d197e7-8539-4735-8df0-2deb84e20cf9-kube-api-access-zdxkz\") pod \"91d197e7-8539-4735-8df0-2deb84e20cf9\" (UID: \"91d197e7-8539-4735-8df0-2deb84e20cf9\") " Nov 02 14:25:39.639546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.639522 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d197e7-8539-4735-8df0-2deb84e20cf9-kube-api-access-zdxkz" pod "91d197e7-8539-4735-8df0-2deb84e20cf9" (UID: "91d197e7-8539-4735-8df0-2deb84e20cf9"). InnerVolumeSpecName "kube-api-access-zdxkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:39.738581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.738508 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxkz\" (UniqueName: \"kubernetes.io/projected/91d197e7-8539-4735-8df0-2deb84e20cf9-kube-api-access-zdxkz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:39.836920 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.836888 4431 patch_prober.go:29] interesting pod/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:25:39.837061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:39.836940 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:25:40.396687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.396614 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-4986cc"] Nov 02 14:25:40.405488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.405461 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-4986cc"] Nov 02 14:25:40.410644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.410610 4431 generic.go:356] "Generic (PLEG): container finished" podID="91d197e7-8539-4735-8df0-2deb84e20cf9" containerID="0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e" exitCode=137 Nov 02 14:25:40.410846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.410694 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:40.410846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.410814 4431 scope.go:122] "RemoveContainer" containerID="0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e" Nov 02 14:25:40.426007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.425987 4431 scope.go:122] "RemoveContainer" containerID="0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e" Nov 02 14:25:40.429467 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:40.428974 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e\": container with ID starting with 0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e not found: ID does not exist" containerID="0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e" Nov 02 14:25:40.429467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.429003 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e"} err="failed to get container status \"0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e\": rpc error: code = NotFound desc = could not find container \"0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e\": container with ID starting with 0ab7c4936519498e9a98a8646397f4aa388f8344abf2fa161aa536904635101e not found: ID does not exist" Nov 02 14:25:40.444110 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:40.444075 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7816-469647\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="91d197e7-8539-4735-8df0-2deb84e20cf9" pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:40.542560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.542534 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7816-f861bc"] Nov 02 14:25:40.543076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.543063 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c4930f88-1289-4622-b085-6560b05affc9" containerName="test-curl-7816-4986cc" Nov 02 14:25:40.543130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.543078 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4930f88-1289-4622-b085-6560b05affc9" containerName="test-curl-7816-4986cc" Nov 02 14:25:40.543197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.543188 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4930f88-1289-4622-b085-6560b05affc9" containerName="test-curl-7816-4986cc" Nov 02 14:25:40.546140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.546121 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-f861bc" Nov 02 14:25:40.560942 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:40.560908 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7816-469647\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="91d197e7-8539-4735-8df0-2deb84e20cf9" pod="opendatahub/test-curl-7816-469647" Nov 02 14:25:40.563245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.563225 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-f861bc"] Nov 02 14:25:40.648490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.648392 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdpzq\" (UniqueName: \"kubernetes.io/projected/2009eb69-e822-4432-a14c-d4799d724edc-kube-api-access-tdpzq\") pod \"test-curl-7816-f861bc\" (UID: \"2009eb69-e822-4432-a14c-d4799d724edc\") " pod="opendatahub/test-curl-7816-f861bc" Nov 02 14:25:40.750204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.750175 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdpzq\" (UniqueName: \"kubernetes.io/projected/2009eb69-e822-4432-a14c-d4799d724edc-kube-api-access-tdpzq\") pod \"test-curl-7816-f861bc\" (UID: \"2009eb69-e822-4432-a14c-d4799d724edc\") " pod="opendatahub/test-curl-7816-f861bc" Nov 02 14:25:40.797715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.797683 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdpzq\" (UniqueName: \"kubernetes.io/projected/2009eb69-e822-4432-a14c-d4799d724edc-kube-api-access-tdpzq\") pod \"test-curl-7816-f861bc\" (UID: \"2009eb69-e822-4432-a14c-d4799d724edc\") " pod="opendatahub/test-curl-7816-f861bc" Nov 02 14:25:40.837309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.837284 4431 patch_prober.go:29] interesting pod/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:25:40.837431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.837329 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:25:40.857457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:40.857434 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-f861bc" Nov 02 14:25:41.099337 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.099298 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d197e7-8539-4735-8df0-2deb84e20cf9" path="/var/lib/kubelet/pods/91d197e7-8539-4735-8df0-2deb84e20cf9/volumes" Nov 02 14:25:41.198603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.198574 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7816-f861bc"] Nov 02 14:25:41.200638 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:41.200610 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2009eb69_e822_4432_a14c_d4799d724edc.slice/crio-9161c9cdf2ab0072fcff18e349e523035b30110e1ec5d1281c9523f1072623c9 WatchSource:0}: Error finding container 9161c9cdf2ab0072fcff18e349e523035b30110e1ec5d1281c9523f1072623c9: Status 404 returned error can't find the container with id 9161c9cdf2ab0072fcff18e349e523035b30110e1ec5d1281c9523f1072623c9 Nov 02 14:25:41.419488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.419456 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-f861bc" event={"ID":"2009eb69-e822-4432-a14c-d4799d724edc","Type":"ContainerStarted","Data":"ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8"} Nov 02 14:25:41.419879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.419504 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7816-f861bc" event={"ID":"2009eb69-e822-4432-a14c-d4799d724edc","Type":"ContainerStarted","Data":"9161c9cdf2ab0072fcff18e349e523035b30110e1ec5d1281c9523f1072623c9"} Nov 02 14:25:41.419879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.419547 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7816-4986cc" podUID="c4930f88-1289-4622-b085-6560b05affc9" containerName="test-curl-7816-4986cc" containerID="cri-o://f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9" gracePeriod=2 Nov 02 14:25:41.462894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.462839 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7816-f861bc" podStartSLOduration=1.462825592 podStartE2EDuration="1.462825592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:41.459690244 +0000 UTC m=+1706.496795178" watchObservedRunningTime="2026-11-02 14:25:41.462825592 +0000 UTC m=+1706.499930559" Nov 02 14:25:41.636692 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.636668 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-f108c6" Nov 02 14:25:41.663317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.663290 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75-kube-api-access-dzs92\" (UniqueName: \"kubernetes.io/projected/8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75-kube-api-access-dzs92\") pod \"8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75\" (UID: \"8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75\") " Nov 02 14:25:41.665288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.665261 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75-kube-api-access-dzs92" pod "8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75" (UID: "8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75"). InnerVolumeSpecName "kube-api-access-dzs92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:41.765288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.765212 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzs92\" (UniqueName: \"kubernetes.io/projected/8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75-kube-api-access-dzs92\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:41.837248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.837220 4431 patch_prober.go:29] interesting pod/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:25:41.837379 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:41.837270 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:25:42.419481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.417160 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7816-f861bc"] Nov 02 14:25:42.421638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.421604 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7816-f861bc"] Nov 02 14:25:42.430436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.430405 4431 generic.go:356] "Generic (PLEG): container finished" podID="8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75" containerID="6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d" exitCode=137 Nov 02 14:25:42.431049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.431030 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-f108c6" Nov 02 14:25:42.432869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.432848 4431 scope.go:122] "RemoveContainer" containerID="6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d" Nov 02 14:25:42.444388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.444368 4431 scope.go:122] "RemoveContainer" containerID="6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d" Nov 02 14:25:42.444647 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:42.444630 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d\": container with ID starting with 6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d not found: ID does not exist" containerID="6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d" Nov 02 14:25:42.444717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.444653 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d"} err="failed to get container status \"6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d\": rpc error: code = NotFound desc = could not find container \"6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d\": container with ID starting with 6bb0012cad3262e4c62ff5354ea939ff0ba05f279e10c4c9338a6cea896c1a3d not found: ID does not exist" Nov 02 14:25:42.456806 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:42.456753 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7816-f108c6\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75" pod="opendatahub/test-curl-7816-f108c6" Nov 02 14:25:42.837436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.837354 4431 patch_prober.go:29] interesting pod/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 02 14:25:42.837436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:42.837408 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 02 14:25:43.099722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.099646 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75" path="/var/lib/kubelet/pods/8b5f4a0f-9ba0-4882-a5ee-6bf1e26c2a75/volumes" Nov 02 14:25:43.440505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.440472 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7816-f861bc" podUID="2009eb69-e822-4432-a14c-d4799d724edc" containerName="test-curl-7816-f861bc" containerID="cri-o://ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8" gracePeriod=2 Nov 02 14:25:43.627342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.627311 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:43.718788 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.718696 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4930f88-1289-4622-b085-6560b05affc9-kube-api-access-hbq6z\" (UniqueName: \"kubernetes.io/projected/c4930f88-1289-4622-b085-6560b05affc9-kube-api-access-hbq6z\") pod \"c4930f88-1289-4622-b085-6560b05affc9\" (UID: \"c4930f88-1289-4622-b085-6560b05affc9\") " Nov 02 14:25:43.720827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.720799 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4930f88-1289-4622-b085-6560b05affc9-kube-api-access-hbq6z" pod "c4930f88-1289-4622-b085-6560b05affc9" (UID: "c4930f88-1289-4622-b085-6560b05affc9"). InnerVolumeSpecName "kube-api-access-hbq6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:43.820626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.820593 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbq6z\" (UniqueName: \"kubernetes.io/projected/c4930f88-1289-4622-b085-6560b05affc9-kube-api-access-hbq6z\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:43.840812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.840790 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:25:43.841246 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.841231 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:25:43.841884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:43.841865 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:25:44.447910 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:44.447877 4431 generic.go:356] "Generic (PLEG): container finished" podID="c4930f88-1289-4622-b085-6560b05affc9" containerID="f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9" exitCode=137 Nov 02 14:25:44.448387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:44.447932 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:44.448387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:44.447970 4431 scope.go:122] "RemoveContainer" containerID="f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9" Nov 02 14:25:44.458053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:44.458040 4431 scope.go:122] "RemoveContainer" containerID="f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9" Nov 02 14:25:44.458317 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:44.458299 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9\": container with ID starting with f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9 not found: ID does not exist" containerID="f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9" Nov 02 14:25:44.458361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:44.458327 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9"} err="failed to get container status \"f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9\": rpc error: code = NotFound desc = could not find container \"f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9\": container with ID starting with f4aae37912407bc0ccd64798adde8007eef6089e4bff9ce0384ab91364fbb1c9 not found: ID does not exist" Nov 02 14:25:44.471796 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:44.471760 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7816-4986cc\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="c4930f88-1289-4622-b085-6560b05affc9" pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:45.076070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:45.076031 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4930f88-1289-4622-b085-6560b05affc9" path="/var/lib/kubelet/pods/c4930f88-1289-4622-b085-6560b05affc9/volumes" Nov 02 14:25:45.086279 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:45.086249 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7816-4986cc\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="c4930f88-1289-4622-b085-6560b05affc9" pod="opendatahub/test-curl-7816-4986cc" Nov 02 14:25:45.499102 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:45.499068 4431 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2009eb69_e822_4432_a14c_d4799d724edc.slice/crio-conmon-ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2009eb69_e822_4432_a14c_d4799d724edc.slice/crio-ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8.scope\": RecentStats: unable to find data in memory cache]" Nov 02 14:25:45.674387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:45.674366 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-f861bc" Nov 02 14:25:45.743868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:45.743769 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2009eb69-e822-4432-a14c-d4799d724edc-kube-api-access-tdpzq\" (UniqueName: \"kubernetes.io/projected/2009eb69-e822-4432-a14c-d4799d724edc-kube-api-access-tdpzq\") pod \"2009eb69-e822-4432-a14c-d4799d724edc\" (UID: \"2009eb69-e822-4432-a14c-d4799d724edc\") " Nov 02 14:25:45.745766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:45.745741 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2009eb69-e822-4432-a14c-d4799d724edc-kube-api-access-tdpzq" pod "2009eb69-e822-4432-a14c-d4799d724edc" (UID: "2009eb69-e822-4432-a14c-d4799d724edc"). InnerVolumeSpecName "kube-api-access-tdpzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:25:45.845400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:45.845380 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdpzq\" (UniqueName: \"kubernetes.io/projected/2009eb69-e822-4432-a14c-d4799d724edc-kube-api-access-tdpzq\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:25:46.464280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:46.464244 4431 generic.go:356] "Generic (PLEG): container finished" podID="2009eb69-e822-4432-a14c-d4799d724edc" containerID="ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8" exitCode=137 Nov 02 14:25:46.464478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:46.464282 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7816-f861bc" Nov 02 14:25:46.464478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:46.464297 4431 scope.go:122] "RemoveContainer" containerID="ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8" Nov 02 14:25:46.473905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:46.473886 4431 scope.go:122] "RemoveContainer" containerID="ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8" Nov 02 14:25:46.474162 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:46.474144 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8\": container with ID starting with ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8 not found: ID does not exist" containerID="ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8" Nov 02 14:25:46.474223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:46.474172 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8"} err="failed to get container status \"ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8\": rpc error: code = NotFound desc = could not find container \"ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8\": container with ID starting with ce5368f275e2b8a014952bda383677b4842acd3abfb1b94205ab1fc34fa1c4b8 not found: ID does not exist" Nov 02 14:25:46.486865 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:46.486840 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7816-f861bc\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="2009eb69-e822-4432-a14c-d4799d724edc" pod="opendatahub/test-curl-7816-f861bc" Nov 02 14:25:47.075830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:47.075798 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2009eb69-e822-4432-a14c-d4799d724edc" path="/var/lib/kubelet/pods/2009eb69-e822-4432-a14c-d4799d724edc/volumes" Nov 02 14:25:51.283683 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:25:51.283635 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:25:53.670791 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.670727 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz"] Nov 02 14:25:53.672243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.672213 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2009eb69-e822-4432-a14c-d4799d724edc" containerName="test-curl-7816-f861bc" Nov 02 14:25:53.672243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.672243 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2009eb69-e822-4432-a14c-d4799d724edc" containerName="test-curl-7816-f861bc" Nov 02 14:25:53.672429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.672411 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2009eb69-e822-4432-a14c-d4799d724edc" containerName="test-curl-7816-f861bc" Nov 02 14:25:53.676043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.676022 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.703752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.703722 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-147981-openshift-default-dockercfg-ghdbv\"" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.708347 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz"] Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709031 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-credential-socket\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709061 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-podinfo\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709083 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-token\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709130 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be03ff8-cbed-47db-8ad1-b071adfb2865-istiod-ca-cert\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709196 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-socket\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709227 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-certs\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709281 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm58v\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-kube-api-access-zm58v\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709318 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-envoy\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.709917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.709344 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-data\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810265 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810238 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-socket\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810280 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-certs\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810329 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm58v\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-kube-api-access-zm58v\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810364 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-envoy\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810392 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-data\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810434 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-credential-socket\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810467 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-podinfo\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810492 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-token\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810577 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be03ff8-cbed-47db-8ad1-b071adfb2865-istiod-ca-cert\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.810744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810689 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-socket\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.811004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.810739 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-certs\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.811169 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.811141 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-data\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.811302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.811265 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-credential-socket\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.811430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.811374 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be03ff8-cbed-47db-8ad1-b071adfb2865-istiod-ca-cert\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.813149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.813123 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-envoy\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.813257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.813183 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-podinfo\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.840001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.839973 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm58v\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-kube-api-access-zm58v\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.841419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.841395 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-token\") pod \"e2e-autores-147981-openshift-default-76d9954467-94hjz\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:53.863103 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.863083 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td"] Nov 02 14:25:53.868433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.868413 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3690233-md7td"] Nov 02 14:25:53.999851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:53.993027 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:54.391487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.391450 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz"] Nov 02 14:25:54.393093 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:54.393072 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be03ff8_cbed_47db_8ad1_b071adfb2865.slice/crio-f1307a8943cddd50ad735438731cc243a81b85372bb0421f8d58afe24ea25582 WatchSource:0}: Error finding container f1307a8943cddd50ad735438731cc243a81b85372bb0421f8d58afe24ea25582: Status 404 returned error can't find the container with id f1307a8943cddd50ad735438731cc243a81b85372bb0421f8d58afe24ea25582 Nov 02 14:25:54.396515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.396472 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:54.396594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.396562 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:54.396673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.396605 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:54.607510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.607473 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" event={"ID":"6be03ff8-cbed-47db-8ad1-b071adfb2865","Type":"ContainerStarted","Data":"9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41"} Nov 02 14:25:54.607806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.607768 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" event={"ID":"6be03ff8-cbed-47db-8ad1-b071adfb2865","Type":"ContainerStarted","Data":"f1307a8943cddd50ad735438731cc243a81b85372bb0421f8d58afe24ea25582"} Nov 02 14:25:54.633524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.633493 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r"] Nov 02 14:25:54.634011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.633712 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" containerID="cri-o://4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05" gracePeriod=30 Nov 02 14:25:54.667325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.667237 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" podStartSLOduration=1.667224022 podStartE2EDuration="1.667224022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:54.663904352 +0000 UTC m=+1719.701009285" watchObservedRunningTime="2026-11-02 14:25:54.667224022 +0000 UTC m=+1719.704328953" Nov 02 14:25:54.994107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:54.994020 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:55.075951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.075911 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e549b7f5-6024-4912-b654-fca247d92fbe" path="/var/lib/kubelet/pods/e549b7f5-6024-4912-b654-fca247d92fbe/volumes" Nov 02 14:25:55.759372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.759325 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9"] Nov 02 14:25:55.763839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.763813 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.793540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.793510 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3261bdf2-openshift-default-dockercfg-8fccl\"" Nov 02 14:25:55.797187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.797164 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9"] Nov 02 14:25:55.840472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840446 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-data\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.840630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840491 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-podinfo\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.840703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840625 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-certs\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.840748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840714 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-credential-socket\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.840815 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840767 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-envoy\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.840865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840833 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-token\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.840865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840858 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-socket\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.840954 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840873 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtqr\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-kube-api-access-7jtqr\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.840992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.840958 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f5d43-c993-449c-a0af-7a250d064b63-istiod-ca-cert\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.943688 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.943520 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-socket\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.943969 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtqr\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-kube-api-access-7jtqr\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.944034 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f5d43-c993-449c-a0af-7a250d064b63-istiod-ca-cert\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.944139 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-data\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.944195 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-podinfo\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.944284 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-certs\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.944358 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-credential-socket\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.944441 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-envoy\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.944492 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-token\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.946849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.946346 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-socket\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.948583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.948561 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-credential-socket\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.949018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.948976 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-certs\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.956918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.950568 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f5d43-c993-449c-a0af-7a250d064b63-istiod-ca-cert\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.956918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.950848 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-data\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.971673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.971642 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-token\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:55.991504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:55.991464 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtqr\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-kube-api-access-7jtqr\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:56.013842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.003728 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-podinfo\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:56.013842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.007973 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-envoy\") pod \"e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:56.020172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.020146 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:56.084553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.084521 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:56.403581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.403554 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9"] Nov 02 14:25:56.405351 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:25:56.405320 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8f5d43_c993_449c_a0af_7a250d064b63.slice/crio-a163c2dd5795e8c6fccf30818adcef31e081d098da2b056350cedd6e920602d3 WatchSource:0}: Error finding container a163c2dd5795e8c6fccf30818adcef31e081d098da2b056350cedd6e920602d3: Status 404 returned error can't find the container with id a163c2dd5795e8c6fccf30818adcef31e081d098da2b056350cedd6e920602d3 Nov 02 14:25:56.408138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.408102 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:56.408266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.408171 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:56.408266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.408202 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:25:56.622499 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.622467 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" event={"ID":"7c8f5d43-c993-449c-a0af-7a250d064b63","Type":"ContainerStarted","Data":"a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214"} Nov 02 14:25:56.622701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.622532 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" event={"ID":"7c8f5d43-c993-449c-a0af-7a250d064b63","Type":"ContainerStarted","Data":"a163c2dd5795e8c6fccf30818adcef31e081d098da2b056350cedd6e920602d3"} Nov 02 14:25:56.622947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.622925 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:56.623996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.623974 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:25:56.672363 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:56.672269 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podStartSLOduration=1.672256781 podStartE2EDuration="1.672256781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:56.671305676 +0000 UTC m=+1721.708410608" watchObservedRunningTime="2026-11-02 14:25:56.672256781 +0000 UTC m=+1721.709361710" Nov 02 14:25:57.034997 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:57.034918 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl"] Nov 02 14:25:57.039272 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:57.039245 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9867ab2a-gfnfl"] Nov 02 14:25:57.076980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:57.076949 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef38aff1-2a98-440c-869a-f6f5570f868b" path="/var/lib/kubelet/pods/ef38aff1-2a98-440c-869a-f6f5570f868b/volumes" Nov 02 14:25:57.085339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:57.085317 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:57.091494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:57.091462 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:57.628853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:57.628822 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:57.629860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:57.629838 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:25:58.843207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:58.843171 4431 patch_prober.go:29] interesting pod/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 02 14:25:58.843620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:58.843232 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:25:59.906480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:25:59.906423 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:26:00.007675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.007643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/181db062-305e-4f1f-942c-3b1feed06efe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181db062-305e-4f1f-942c-3b1feed06efe-istio-podinfo\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.007843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.007705 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-envoy\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.007843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.007815 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-certs\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.007985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.007858 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-kube-api-access-hg2c7\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-kube-api-access-hg2c7\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.007985 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.007976 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-istio-token\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-istio-token\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.008095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008003 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181db062-305e-4f1f-942c-3b1feed06efe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181db062-305e-4f1f-942c-3b1feed06efe-istiod-ca-cert\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.008154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008103 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-socket\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.008154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008123 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-certs" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:00.008154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008146 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-credential-socket\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.008316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008193 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-data\") pod \"181db062-305e-4f1f-942c-3b1feed06efe\" (UID: \"181db062-305e-4f1f-942c-3b1feed06efe\") " Nov 02 14:26:00.008454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008434 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181db062-305e-4f1f-942c-3b1feed06efe-istiod-ca-cert" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:00.008716 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008694 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-data" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:00.008835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008711 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.008835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008732 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181db062-305e-4f1f-942c-3b1feed06efe-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.008835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008763 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-credential-socket" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:00.009027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.008835 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-socket" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:00.010411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.010387 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-envoy" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:26:00.010728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.010707 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/181db062-305e-4f1f-942c-3b1feed06efe-istio-podinfo" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:26:00.010921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.010896 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-kube-api-access-hg2c7" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "kube-api-access-hg2c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:00.011039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.010960 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-istio-token" pod "181db062-305e-4f1f-942c-3b1feed06efe" (UID: "181db062-305e-4f1f-942c-3b1feed06efe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:00.110105 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.110079 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.110105 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.110107 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.110283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.110121 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.110283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.110131 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.110283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.110142 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181db062-305e-4f1f-942c-3b1feed06efe-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.110283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.110150 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181db062-305e-4f1f-942c-3b1feed06efe-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.110283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.110158 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2c7\" (UniqueName: \"kubernetes.io/projected/181db062-305e-4f1f-942c-3b1feed06efe-kube-api-access-hg2c7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:00.649343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.649306 4431 generic.go:356] "Generic (PLEG): container finished" podID="181db062-305e-4f1f-942c-3b1feed06efe" containerID="4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05" exitCode=0 Nov 02 14:26:00.649489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.649368 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" Nov 02 14:26:00.649489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.649398 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" event={"ID":"181db062-305e-4f1f-942c-3b1feed06efe","Type":"ContainerDied","Data":"4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05"} Nov 02 14:26:00.649489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.649472 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r" event={"ID":"181db062-305e-4f1f-942c-3b1feed06efe","Type":"ContainerDied","Data":"b6b9e3ecdf806664ba4013cdb7b25f6314bc86a2d858cc60ed640795df17fc79"} Nov 02 14:26:00.649602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.649496 4431 scope.go:122] "RemoveContainer" containerID="4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05" Nov 02 14:26:00.660434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.660417 4431 scope.go:122] "RemoveContainer" containerID="fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2" Nov 02 14:26:00.669280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.669262 4431 scope.go:122] "RemoveContainer" containerID="4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05" Nov 02 14:26:00.669542 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:26:00.669524 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05\": container with ID starting with 4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05 not found: ID does not exist" containerID="4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05" Nov 02 14:26:00.669603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.669550 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05"} err="failed to get container status \"4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05\": rpc error: code = NotFound desc = could not find container \"4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05\": container with ID starting with 4ff5329e8a6721880aeb431563d687035c0a1c4c77bcd14d9ec73b17e659bf05 not found: ID does not exist" Nov 02 14:26:00.669603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.669568 4431 scope.go:122] "RemoveContainer" containerID="fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2" Nov 02 14:26:00.669848 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:26:00.669827 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2\": container with ID starting with fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2 not found: ID does not exist" containerID="fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2" Nov 02 14:26:00.669906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.669858 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2"} err="failed to get container status \"fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2\": rpc error: code = NotFound desc = could not find container \"fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2\": container with ID starting with fc1e7fc8da1c9f635379a525a5e110dad4381f713c79c14bd01c3a4a432ea2f2 not found: ID does not exist" Nov 02 14:26:00.740940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.740919 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r"] Nov 02 14:26:00.754061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:00.754035 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b3690233-openshift-default-7bcfb888c7-tcb4r"] Nov 02 14:26:01.075714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:01.075644 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181db062-305e-4f1f-942c-3b1feed06efe" path="/var/lib/kubelet/pods/181db062-305e-4f1f-942c-3b1feed06efe/volumes" Nov 02 14:26:03.119673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:03.119636 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:26:07.126154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:07.126123 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8"] Nov 02 14:26:12.127034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.126986 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" podUID="93655790-866b-4892-a19c-129b9365f3d0" containerName="payload-processing" containerID="cri-o://28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283" gracePeriod=25 Nov 02 14:26:12.398441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.398417 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:26:12.445077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.445051 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-provider-credentials\") pod \"93655790-866b-4892-a19c-129b9365f3d0\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " Nov 02 14:26:12.445231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.445093 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-kube-api-access-g46hw\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-kube-api-access-g46hw\") pod \"93655790-866b-4892-a19c-129b9365f3d0\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " Nov 02 14:26:12.445231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.445122 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-routing-overlay-volume\") pod \"93655790-866b-4892-a19c-129b9365f3d0\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " Nov 02 14:26:12.445231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.445170 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-plugins-config-volume\") pod \"93655790-866b-4892-a19c-129b9365f3d0\" (UID: \"93655790-866b-4892-a19c-129b9365f3d0\") " Nov 02 14:26:12.445517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.445490 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-routing-overlay-volume" pod "93655790-866b-4892-a19c-129b9365f3d0" (UID: "93655790-866b-4892-a19c-129b9365f3d0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:12.445655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.445600 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-plugins-config-volume" pod "93655790-866b-4892-a19c-129b9365f3d0" (UID: "93655790-866b-4892-a19c-129b9365f3d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:12.445750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.445734 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:12.445814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.445754 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93655790-866b-4892-a19c-129b9365f3d0-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:12.447316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.447288 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-provider-credentials" pod "93655790-866b-4892-a19c-129b9365f3d0" (UID: "93655790-866b-4892-a19c-129b9365f3d0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:12.447428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.447406 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-kube-api-access-g46hw" pod "93655790-866b-4892-a19c-129b9365f3d0" (UID: "93655790-866b-4892-a19c-129b9365f3d0"). InnerVolumeSpecName "kube-api-access-g46hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:12.546622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.546559 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:12.546622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.546585 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g46hw\" (UniqueName: \"kubernetes.io/projected/93655790-866b-4892-a19c-129b9365f3d0-kube-api-access-g46hw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:12.731636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.731600 4431 generic.go:356] "Generic (PLEG): container finished" podID="93655790-866b-4892-a19c-129b9365f3d0" containerID="28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283" exitCode=0 Nov 02 14:26:12.731915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.731648 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" Nov 02 14:26:12.731915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.731681 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" event={"ID":"93655790-866b-4892-a19c-129b9365f3d0","Type":"ContainerDied","Data":"28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283"} Nov 02 14:26:12.731915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.731738 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8" event={"ID":"93655790-866b-4892-a19c-129b9365f3d0","Type":"ContainerDied","Data":"5eaa138241168a283cab0c997592a4a41fc51a3be1e533f62af65dbe87975583"} Nov 02 14:26:12.731915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.731760 4431 scope.go:122] "RemoveContainer" containerID="28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283" Nov 02 14:26:12.742198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.742178 4431 scope.go:122] "RemoveContainer" containerID="28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283" Nov 02 14:26:12.742444 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:26:12.742424 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283\": container with ID starting with 28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283 not found: ID does not exist" containerID="28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283" Nov 02 14:26:12.742505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.742453 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283"} err="failed to get container status \"28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283\": rpc error: code = NotFound desc = could not find container \"28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283\": container with ID starting with 28a0d18ad0cc80d7474ecac091301e6ad2cb5f16a5a4d7660985cc849d27f283 not found: ID does not exist" Nov 02 14:26:12.803560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.803536 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8"] Nov 02 14:26:12.816906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:12.816882 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5dfcgcwt8"] Nov 02 14:26:13.075792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:13.075708 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93655790-866b-4892-a19c-129b9365f3d0" path="/var/lib/kubelet/pods/93655790-866b-4892-a19c-129b9365f3d0/volumes" Nov 02 14:26:17.058638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:17.058600 4431 scope.go:122] "RemoveContainer" containerID="b86c04d23a861a5959952c200d5e9e6b5d5a4841297b17a315c1777c7dc8f247" Nov 02 14:26:17.071313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:17.071195 4431 scope.go:122] "RemoveContainer" containerID="83eae1dd9be85d4fa09507633084ed920ae20a2ffa68504f4241caa19628e520" Nov 02 14:26:17.081295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:17.081273 4431 scope.go:122] "RemoveContainer" containerID="c45b6fb8e4797d37792d175ae7fbe4fa37095c3aa903098f04a2b6075656903b" Nov 02 14:26:17.089811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:17.089794 4431 scope.go:122] "RemoveContainer" containerID="5bb5782ecc18602f6e98e6019861e44442c5b3dc13bd5fd814765326f541ca28" Nov 02 14:26:21.283761 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:26:21.283719 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:26:23.208896 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:23.208858 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:26:27.553719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.553677 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9"] Nov 02 14:26:27.554333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.554314 4431 cpu_manager.go:414] "containerMap: removing container" podUID="93655790-866b-4892-a19c-129b9365f3d0" containerName="payload-processing" Nov 02 14:26:27.554380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.554335 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93655790-866b-4892-a19c-129b9365f3d0" containerName="payload-processing" Nov 02 14:26:27.554380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.554355 4431 cpu_manager.go:414] "containerMap: removing container" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" Nov 02 14:26:27.554380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.554360 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" Nov 02 14:26:27.554478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.554473 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" Nov 02 14:26:27.554513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.554482 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="93655790-866b-4892-a19c-129b9365f3d0" containerName="payload-processing" Nov 02 14:26:27.554513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.554490 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" Nov 02 14:26:27.558177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.558156 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.615802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.593009 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-8434b5bb1061\"" Nov 02 14:26:27.615802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.593111 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"openshift-service-ca.crt\"" Nov 02 14:26:27.615802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.593184 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-e2e--4e96fc15-dockercfg-gffbp\"" Nov 02 14:26:27.615802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.593225 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"e2e-cfl-622143-provider-cred\"" Nov 02 14:26:27.626803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.618179 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9"] Nov 02 14:26:27.627154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.627130 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-ca19ab63a2a2\"" Nov 02 14:26:27.627352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.627339 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"routing-overlay\"" Nov 02 14:26:27.627499 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.627487 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"kube-root-ca.crt\"" Nov 02 14:26:27.635506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.628293 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-plugins-e2e-extmod-a-944506\"" Nov 02 14:26:27.637667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.637260 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.637967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.637945 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24jv\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-kube-api-access-c24jv\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.638227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.638206 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.638581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.638562 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.740723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.740652 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.740723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.740702 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24jv\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-kube-api-access-c24jv\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.740975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.740806 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.740975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.740963 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.741426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.741404 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.741530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.741512 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.743437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.743417 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.775154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.775134 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24jv\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-kube-api-access-c24jv\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:27.871356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:27.871215 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:28.209676 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.209653 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9"] Nov 02 14:26:28.211600 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:26:28.211573 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107ce5f9_75c3_4a68_b9ee_8ea4e34bb3f7.slice/crio-6029520a04095cacbe188650169ab67c94fcbbde9dd96011c8cf6094bade17a9 WatchSource:0}: Error finding container 6029520a04095cacbe188650169ab67c94fcbbde9dd96011c8cf6094bade17a9: Status 404 returned error can't find the container with id 6029520a04095cacbe188650169ab67c94fcbbde9dd96011c8cf6094bade17a9 Nov 02 14:26:28.499806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.499705 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9"] Nov 02 14:26:28.537089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.537049 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7"] Nov 02 14:26:28.537762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.537738 4431 cpu_manager.go:414] "containerMap: removing container" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" Nov 02 14:26:28.537762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.537761 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181db062-305e-4f1f-942c-3b1feed06efe" containerName="istio-proxy" Nov 02 14:26:28.540262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.540241 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.558025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.558005 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-0a71f3a9867d\"" Nov 02 14:26:28.565837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.565816 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7"] Nov 02 14:26:28.657120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.657085 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.657290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.657243 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.657290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.657275 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.657434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.657313 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ftt\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-kube-api-access-j8ftt\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.759099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.759031 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.759099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.759070 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.759268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.759115 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ftt\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-kube-api-access-j8ftt\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.759306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.759262 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.759725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.759704 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.759882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.759859 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.761890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.761867 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.785827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.785807 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ftt\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-kube-api-access-j8ftt\") pod \"payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.853584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.853553 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:28.857217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.857193 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" event={"ID":"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7","Type":"ContainerStarted","Data":"2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833"} Nov 02 14:26:28.857332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.857252 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" event={"ID":"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7","Type":"ContainerStarted","Data":"6029520a04095cacbe188650169ab67c94fcbbde9dd96011c8cf6094bade17a9"} Nov 02 14:26:28.857429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.857408 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:28.904648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:28.903333 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" podStartSLOduration=1.903314007 podStartE2EDuration="1.903314007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:28.901682974 +0000 UTC m=+1753.938787906" watchObservedRunningTime="2026-11-02 14:26:28.903314007 +0000 UTC m=+1753.940418941" Nov 02 14:26:29.163384 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:29.163358 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7"] Nov 02 14:26:29.164374 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:26:29.164350 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9eafd5_2b84_4dfb_8b07_d8be5e88857e.slice/crio-049ea2c73e03254edd3b4e2cac88594670acdc119e018a14bab1bc3eeead30b7 WatchSource:0}: Error finding container 049ea2c73e03254edd3b4e2cac88594670acdc119e018a14bab1bc3eeead30b7: Status 404 returned error can't find the container with id 049ea2c73e03254edd3b4e2cac88594670acdc119e018a14bab1bc3eeead30b7 Nov 02 14:26:29.864832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:29.864801 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" event={"ID":"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e","Type":"ContainerStarted","Data":"29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec"} Nov 02 14:26:29.865232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:29.864853 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" event={"ID":"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e","Type":"ContainerStarted","Data":"049ea2c73e03254edd3b4e2cac88594670acdc119e018a14bab1bc3eeead30b7"} Nov 02 14:26:29.865232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:29.864876 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:29.913462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:29.913408 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" podStartSLOduration=1.91339444 podStartE2EDuration="1.91339444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:29.911641266 +0000 UTC m=+1754.948746198" watchObservedRunningTime="2026-11-02 14:26:29.91339444 +0000 UTC m=+1754.950499372" Nov 02 14:26:34.865976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:34.865914 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" podUID="107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" containerName="payload-processing" containerID="cri-o://2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833" gracePeriod=25 Nov 02 14:26:35.122882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.122822 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:35.235150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.235116 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-plugins-config-volume\") pod \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " Nov 02 14:26:35.235332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.235246 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-routing-overlay-volume\") pod \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " Nov 02 14:26:35.235332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.235297 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-provider-credentials\") pod \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " Nov 02 14:26:35.235459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.235385 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-kube-api-access-c24jv\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-kube-api-access-c24jv\") pod \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\" (UID: \"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7\") " Nov 02 14:26:35.235607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.235575 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-plugins-config-volume" pod "107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" (UID: "107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:35.235664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.235599 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-routing-overlay-volume" pod "107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" (UID: "107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:26:35.235874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.235854 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:35.235874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.235875 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:35.237496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.237467 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-kube-api-access-c24jv" pod "107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" (UID: "107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7"). InnerVolumeSpecName "kube-api-access-c24jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:35.237496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.237485 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-provider-credentials" pod "107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" (UID: "107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:26:35.336729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.336696 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:35.336729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.336725 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c24jv\" (UniqueName: \"kubernetes.io/projected/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7-kube-api-access-c24jv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:26:35.906171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.906136 4431 generic.go:356] "Generic (PLEG): container finished" podID="107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" containerID="2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833" exitCode=0 Nov 02 14:26:35.906583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.906177 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" Nov 02 14:26:35.906583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.906218 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" event={"ID":"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7","Type":"ContainerDied","Data":"2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833"} Nov 02 14:26:35.906583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.906275 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9" event={"ID":"107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7","Type":"ContainerDied","Data":"6029520a04095cacbe188650169ab67c94fcbbde9dd96011c8cf6094bade17a9"} Nov 02 14:26:35.906583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.906307 4431 scope.go:122] "RemoveContainer" containerID="2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833" Nov 02 14:26:35.916664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.916649 4431 scope.go:122] "RemoveContainer" containerID="2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833" Nov 02 14:26:35.919142 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:26:35.919116 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833\": container with ID starting with 2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833 not found: ID does not exist" containerID="2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833" Nov 02 14:26:35.919244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.919147 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833"} err="failed to get container status \"2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833\": rpc error: code = NotFound desc = could not find container \"2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833\": container with ID starting with 2b9ce4606c7770f7048631811d4876eaa05e749c27996c34063086af40e0f833 not found: ID does not exist" Nov 02 14:26:35.963608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.963578 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9"] Nov 02 14:26:35.969658 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:35.969636 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578nwlq9"] Nov 02 14:26:37.075282 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:37.075253 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" path="/var/lib/kubelet/pods/107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7/volumes" Nov 02 14:26:40.913801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:40.908991 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:26:49.012558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.012524 4431 generic.go:356] "Generic (PLEG): container finished" podID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerID="a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214" exitCode=0 Nov 02 14:26:49.012558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.012550 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" event={"ID":"7c8f5d43-c993-449c-a0af-7a250d064b63","Type":"ContainerDied","Data":"a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214"} Nov 02 14:26:49.013135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.012993 4431 scope.go:122] "RemoveContainer" containerID="a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214" Nov 02 14:26:49.014597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.014565 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:26:49.014707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.014635 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:26:49.014707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.014669 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:26:49.046340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.046310 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj"] Nov 02 14:26:49.049135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.049110 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fcdadf-gdhlj"] Nov 02 14:26:49.078506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:49.078476 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540f1c37-7470-43e5-9f55-6b84b0d5ad18" path="/var/lib/kubelet/pods/540f1c37-7470-43e5-9f55-6b84b0d5ad18/volumes" Nov 02 14:26:50.020960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.020929 4431 generic.go:356] "Generic (PLEG): container finished" podID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerID="9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41" exitCode=0 Nov 02 14:26:50.021453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.021000 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" event={"ID":"6be03ff8-cbed-47db-8ad1-b071adfb2865","Type":"ContainerDied","Data":"9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41"} Nov 02 14:26:50.021453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.021448 4431 scope.go:122] "RemoveContainer" containerID="9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41" Nov 02 14:26:50.023211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.023179 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:26:50.023330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.023208 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" event={"ID":"7c8f5d43-c993-449c-a0af-7a250d064b63","Type":"ContainerStarted","Data":"3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7"} Nov 02 14:26:50.023330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.023251 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:26:50.023330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.023282 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:26:50.092889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.092838 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:26:50.102802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.093961 4431 patch_prober.go:29] interesting pod/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:26:50.102802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:50.094014 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:26:51.031687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:51.031652 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" event={"ID":"6be03ff8-cbed-47db-8ad1-b071adfb2865","Type":"ContainerStarted","Data":"73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33"} Nov 02 14:26:51.092804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:51.090898 4431 patch_prober.go:29] interesting pod/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:26:51.092804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:51.090973 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:26:51.331204 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:26:51.331124 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:26:52.004895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:52.001141 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:26:52.004895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:52.001265 4431 patch_prober.go:29] interesting pod/e2e-autores-147981-openshift-default-76d9954467-94hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 02 14:26:52.004895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:52.001308 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 02 14:26:52.091800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:52.091177 4431 patch_prober.go:29] interesting pod/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:26:52.091800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:52.091239 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:26:52.994440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:52.994407 4431 patch_prober.go:29] interesting pod/e2e-autores-147981-openshift-default-76d9954467-94hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 02 14:26:52.994738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:52.994707 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 02 14:26:53.096801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:53.089879 4431 patch_prober.go:29] interesting pod/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:26:53.096801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:53.089938 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:26:54.000802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:53.998885 4431 patch_prober.go:29] interesting pod/e2e-autores-147981-openshift-default-76d9954467-94hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 02 14:26:54.000802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:53.998946 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 02 14:26:54.088606 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:54.088176 4431 patch_prober.go:29] interesting pod/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:26:54.088606 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:54.088236 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:26:54.997801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:54.994266 4431 patch_prober.go:29] interesting pod/e2e-autores-147981-openshift-default-76d9954467-94hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 02 14:26:54.997801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:54.994343 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 02 14:26:55.085138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.085107 4431 patch_prober.go:29] interesting pod/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 02 14:26:55.085389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.085367 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 02 14:26:55.405970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.405370 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx"] Nov 02 14:26:55.406326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.406303 4431 cpu_manager.go:414] "containerMap: removing container" podUID="107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" containerName="payload-processing" Nov 02 14:26:55.406391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.406327 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" containerName="payload-processing" Nov 02 14:26:55.406528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.406514 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="107ce5f9-75c3-4a68-b9ee-8ea4e34bb3f7" containerName="payload-processing" Nov 02 14:26:55.410827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.410805 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.422837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.420046 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx"] Nov 02 14:26:55.448598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.448569 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-147981\"/\"autores-model-147981-kserve-self-signed-certs\"" Nov 02 14:26:55.448732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.448615 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-147981\"/\"default-dockercfg-q7nmr\"" Nov 02 14:26:55.496731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.496093 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-147981\"/\"kube-root-ca.crt\"" Nov 02 14:26:55.496731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.496215 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-147981\"/\"openshift-service-ca.crt\"" Nov 02 14:26:55.497024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.496982 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-home\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.497089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.497035 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf98c\" (UniqueName: \"kubernetes.io/projected/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-kube-api-access-lf98c\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.497152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.497113 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-model-cache\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.497152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.497138 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tmp-dir\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.497325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.497240 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-dshm\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.497325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.497300 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tls-certs\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.605800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.605467 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-model-cache\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.605800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.605523 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tmp-dir\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.605800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.605759 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-dshm\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.606135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.605844 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tls-certs\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.606135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.606108 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-home\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.606255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.606180 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf98c\" (UniqueName: \"kubernetes.io/projected/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-kube-api-access-lf98c\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.607119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.607089 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-model-cache\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.608281 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.607883 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tmp-dir\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.608680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.608660 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-home\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.612739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.612717 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tls-certs\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.613654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.613618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-dshm\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.631005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.630986 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf98c\" (UniqueName: \"kubernetes.io/projected/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-kube-api-access-lf98c\") pod \"autores-model-147981-kserve-5fdd77f59c-s57cx\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:55.745007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:55.744912 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:56.010853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:56.007726 4431 patch_prober.go:29] interesting pod/e2e-autores-147981-openshift-default-76d9954467-94hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 02 14:26:56.010853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:56.007834 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 02 14:26:56.177989 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:56.177946 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:26:56.178470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:56.178341 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:26:56.179972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:56.179771 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:26:56.305787 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:56.305690 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx"] Nov 02 14:26:56.311335 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:26:56.311302 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79eb61e4_1bbd_4465_9b5b_85b8e112f12c.slice/crio-4c1cfb88879a93d6bac963dafec301c0cd9618e3a8aff12f1e93729867b5353b WatchSource:0}: Error finding container 4c1cfb88879a93d6bac963dafec301c0cd9618e3a8aff12f1e93729867b5353b: Status 404 returned error can't find the container with id 4c1cfb88879a93d6bac963dafec301c0cd9618e3a8aff12f1e93729867b5353b Nov 02 14:26:56.623447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:56.623372 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:26:56.999184 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:56.999156 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:26:57.087371 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:57.087338 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" event={"ID":"79eb61e4-1bbd-4465-9b5b-85b8e112f12c","Type":"ContainerStarted","Data":"2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526"} Nov 02 14:26:57.087848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:57.087391 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" event={"ID":"79eb61e4-1bbd-4465-9b5b-85b8e112f12c","Type":"ContainerStarted","Data":"4c1cfb88879a93d6bac963dafec301c0cd9618e3a8aff12f1e93729867b5353b"} Nov 02 14:26:57.088829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:57.088807 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:26:57.135457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:57.135411 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" podStartSLOduration=2.135395914 podStartE2EDuration="2.135395914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:57.134743517 +0000 UTC m=+1782.171848460" watchObservedRunningTime="2026-11-02 14:26:57.135395914 +0000 UTC m=+1782.172500846" Nov 02 14:26:57.745637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:57.745604 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:26:59.625665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.625632 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88"] Nov 02 14:26:59.628676 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.628660 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.650798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.646345 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:26:59.650798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.646592 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:26:59.650798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.650653 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88"] Nov 02 14:26:59.773715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.773681 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.773897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.773838 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.774080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.774052 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8gq\" (UniqueName: \"kubernetes.io/projected/186f5f0a-523a-454e-9f1c-dac4702ede2e-kube-api-access-fx8gq\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.875394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.875366 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.875565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.875418 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.875565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.875511 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8gq\" (UniqueName: \"kubernetes.io/projected/186f5f0a-523a-454e-9f1c-dac4702ede2e-kube-api-access-fx8gq\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.876136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.876074 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.877746 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.877727 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.903155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.903128 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8gq\" (UniqueName: \"kubernetes.io/projected/186f5f0a-523a-454e-9f1c-dac4702ede2e-kube-api-access-fx8gq\") pod \"maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:26:59.942089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:26:59.942065 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:27:00.261199 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:27:00.261164 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186f5f0a_523a_454e_9f1c_dac4702ede2e.slice/crio-e0b905f611c260f0861f7c7c64c56f7ffd0bd0529ce746fbf06b72cf83e5db02 WatchSource:0}: Error finding container e0b905f611c260f0861f7c7c64c56f7ffd0bd0529ce746fbf06b72cf83e5db02: Status 404 returned error can't find the container with id e0b905f611c260f0861f7c7c64c56f7ffd0bd0529ce746fbf06b72cf83e5db02 Nov 02 14:27:00.263108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:00.263024 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88"] Nov 02 14:27:01.113740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:01.113705 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" event={"ID":"186f5f0a-523a-454e-9f1c-dac4702ede2e","Type":"ContainerStarted","Data":"de64518cd1c50d7522658ea1c25a227932f687e7b16aab4994d15cefe0333207"} Nov 02 14:27:01.114147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:01.113752 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" event={"ID":"186f5f0a-523a-454e-9f1c-dac4702ede2e","Type":"ContainerStarted","Data":"e0b905f611c260f0861f7c7c64c56f7ffd0bd0529ce746fbf06b72cf83e5db02"} Nov 02 14:27:01.160243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:01.160197 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" podStartSLOduration=2.160184876 podStartE2EDuration="2.160184876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:01.158342496 +0000 UTC m=+1786.195447431" watchObservedRunningTime="2026-11-02 14:27:01.160184876 +0000 UTC m=+1786.197289808" Nov 02 14:27:02.122286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:02.122249 4431 generic.go:356] "Generic (PLEG): container finished" podID="186f5f0a-523a-454e-9f1c-dac4702ede2e" containerID="de64518cd1c50d7522658ea1c25a227932f687e7b16aab4994d15cefe0333207" exitCode=0 Nov 02 14:27:02.122686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:02.122328 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" event={"ID":"186f5f0a-523a-454e-9f1c-dac4702ede2e","Type":"ContainerDied","Data":"de64518cd1c50d7522658ea1c25a227932f687e7b16aab4994d15cefe0333207"} Nov 02 14:27:03.287046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.287021 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:27:03.417911 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.417877 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186f5f0a-523a-454e-9f1c-dac4702ede2e-kube-api-access-fx8gq\" (UniqueName: \"kubernetes.io/projected/186f5f0a-523a-454e-9f1c-dac4702ede2e-kube-api-access-fx8gq\") pod \"186f5f0a-523a-454e-9f1c-dac4702ede2e\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " Nov 02 14:27:03.418114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.418004 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-openshift-ca\") pod \"186f5f0a-523a-454e-9f1c-dac4702ede2e\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " Nov 02 14:27:03.418186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.418173 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-xks-ca\") pod \"186f5f0a-523a-454e-9f1c-dac4702ede2e\" (UID: \"186f5f0a-523a-454e-9f1c-dac4702ede2e\") " Nov 02 14:27:03.418605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.418575 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-openshift-ca" pod "186f5f0a-523a-454e-9f1c-dac4702ede2e" (UID: "186f5f0a-523a-454e-9f1c-dac4702ede2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:03.418751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.418737 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:03.420480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.420453 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-xks-ca" pod "186f5f0a-523a-454e-9f1c-dac4702ede2e" (UID: "186f5f0a-523a-454e-9f1c-dac4702ede2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:27:03.420598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.420562 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186f5f0a-523a-454e-9f1c-dac4702ede2e-kube-api-access-fx8gq" pod "186f5f0a-523a-454e-9f1c-dac4702ede2e" (UID: "186f5f0a-523a-454e-9f1c-dac4702ede2e"). InnerVolumeSpecName "kube-api-access-fx8gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:03.520536 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.520500 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8gq\" (UniqueName: \"kubernetes.io/projected/186f5f0a-523a-454e-9f1c-dac4702ede2e-kube-api-access-fx8gq\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:03.520536 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:03.520533 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186f5f0a-523a-454e-9f1c-dac4702ede2e-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:04.135675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:04.135649 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" Nov 02 14:27:04.135891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:04.135645 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88" event={"ID":"186f5f0a-523a-454e-9f1c-dac4702ede2e","Type":"ContainerDied","Data":"e0b905f611c260f0861f7c7c64c56f7ffd0bd0529ce746fbf06b72cf83e5db02"} Nov 02 14:27:04.135891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:04.135814 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0b905f611c260f0861f7c7c64c56f7ffd0bd0529ce746fbf06b72cf83e5db02" Nov 02 14:27:05.745375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:05.745341 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:27:05.758077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:05.758048 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:27:06.036611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:06.036523 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9"] Nov 02 14:27:06.039567 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:06.039543 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-211b8cf5-cbgj9"] Nov 02 14:27:06.069488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:06.069465 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:27:06.166663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:06.166637 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:27:07.078376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:07.078335 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1073b3b8-d6e4-4ef7-a1fd-2c343e47af64" path="/var/lib/kubelet/pods/1073b3b8-d6e4-4ef7-a1fd-2c343e47af64/volumes" Nov 02 14:27:16.233053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:16.233015 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx"] Nov 02 14:27:17.039261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:17.039229 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l"] Nov 02 14:27:17.042895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:17.042870 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f990fc-sdv8l"] Nov 02 14:27:17.076772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:17.076741 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9d2f1d-8c50-4e32-be77-e2e92787c597" path="/var/lib/kubelet/pods/5e9d2f1d-8c50-4e32-be77-e2e92787c597/volumes" Nov 02 14:27:17.155670 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:17.155638 4431 scope.go:122] "RemoveContainer" containerID="9cf551715f6c97b36208ac190e57606d1e5592187c6d2b44c738ef405e5c7af3" Nov 02 14:27:21.285757 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:21.285722 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:27:22.461940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:22.461908 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88"] Nov 02 14:27:22.474487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:22.474461 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3261bdf2-dwq88"] Nov 02 14:27:22.670616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:22.670583 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9"] Nov 02 14:27:22.670876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:22.670838 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" containerID="cri-o://3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7" gracePeriod=30 Nov 02 14:27:23.076323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:23.076290 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186f5f0a-523a-454e-9f1c-dac4702ede2e" path="/var/lib/kubelet/pods/186f5f0a-523a-454e-9f1c-dac4702ede2e/volumes" Nov 02 14:27:24.523387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.523344 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p"] Nov 02 14:27:24.524479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.524451 4431 cpu_manager.go:414] "containerMap: removing container" podUID="186f5f0a-523a-454e-9f1c-dac4702ede2e" containerName="revoke-keys" Nov 02 14:27:24.524479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.524477 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186f5f0a-523a-454e-9f1c-dac4702ede2e" containerName="revoke-keys" Nov 02 14:27:24.524677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.524642 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="186f5f0a-523a-454e-9f1c-dac4702ede2e" containerName="revoke-keys" Nov 02 14:27:24.528715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.528689 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.550941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.550910 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f72bc6-openshift-default-dockercfg-p5hdl\"" Nov 02 14:27:24.564077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.564050 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p"] Nov 02 14:27:24.583454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583414 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-envoy\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.583605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583482 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-token\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.583605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583533 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.583605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583575 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-certs\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.583836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583667 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-podinfo\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.583836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583712 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-data\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.583836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583742 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-credential-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.584017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583845 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qvz\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-kube-api-access-d4qvz\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.584017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.583904 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff47094d-1dd4-4433-aaba-5dd88865d905-istiod-ca-cert\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.685602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.685569 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-podinfo\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.685839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.685617 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-data\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.685839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.685639 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-credential-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.685839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.685711 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qvz\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-kube-api-access-d4qvz\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.685839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.685763 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff47094d-1dd4-4433-aaba-5dd88865d905-istiod-ca-cert\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.685882 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-envoy\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.685928 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-token\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.685977 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.686020 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-certs\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686304 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.686098 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-credential-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.686351 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-certs\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.686377 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-socket\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.686456 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-data\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.686549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.686533 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff47094d-1dd4-4433-aaba-5dd88865d905-istiod-ca-cert\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.688168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.688141 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-envoy\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.688285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.688244 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-podinfo\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.713593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.713537 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qvz\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-kube-api-access-d4qvz\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.714232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.714189 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-token\") pod \"e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:24.844145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:24.844059 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:25.173988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:25.173957 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p"] Nov 02 14:27:25.174121 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:27:25.174056 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff47094d_1dd4_4433_aaba_5dd88865d905.slice/crio-938859ab3e4414090903387a6d95f43994eea3fb6f8ad6bbff2394901443ff0b WatchSource:0}: Error finding container 938859ab3e4414090903387a6d95f43994eea3fb6f8ad6bbff2394901443ff0b: Status 404 returned error can't find the container with id 938859ab3e4414090903387a6d95f43994eea3fb6f8ad6bbff2394901443ff0b Nov 02 14:27:25.177324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:25.177286 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:27:25.177424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:25.177380 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:27:25.177464 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:25.177425 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:27:25.300412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:25.300370 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" event={"ID":"ff47094d-1dd4-4433-aaba-5dd88865d905","Type":"ContainerStarted","Data":"25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c"} Nov 02 14:27:25.300572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:25.300427 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" event={"ID":"ff47094d-1dd4-4433-aaba-5dd88865d905","Type":"ContainerStarted","Data":"938859ab3e4414090903387a6d95f43994eea3fb6f8ad6bbff2394901443ff0b"} Nov 02 14:27:25.350405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:25.350340 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" podStartSLOduration=1.350321597 podStartE2EDuration="1.350321597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:25.348518617 +0000 UTC m=+1810.385623550" watchObservedRunningTime="2026-11-02 14:27:25.350321597 +0000 UTC m=+1810.387426532" Nov 02 14:27:25.845039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:25.845008 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:26.181314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:26.181272 4431 patch_prober.go:29] interesting pod/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:27:26.181499 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:26.181334 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:27:26.850137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:26.850108 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:27.313489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:27.313458 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:27.314424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:27.314407 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:27:27.970325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:27.970302 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:27:28.060799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.055353 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-envoy\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.060799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.055445 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c8f5d43-c993-449c-a0af-7a250d064b63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f5d43-c993-449c-a0af-7a250d064b63-istiod-ca-cert\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.060799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.055632 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-token\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.064798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.061966 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8f5d43-c993-449c-a0af-7a250d064b63-istiod-ca-cert" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:28.064798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.055771 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-socket\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.064798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.062397 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-kube-api-access-7jtqr\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-kube-api-access-7jtqr\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.064798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.062456 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-podinfo\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.064798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.062509 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-certs\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.064798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.062542 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-credential-socket\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.064798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.062604 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-data\") pod \"7c8f5d43-c993-449c-a0af-7a250d064b63\" (UID: \"7c8f5d43-c993-449c-a0af-7a250d064b63\") " Nov 02 14:27:28.064798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.063278 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f5d43-c993-449c-a0af-7a250d064b63-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.102795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.089251 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-data" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:28.107291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.106114 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-socket" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:28.113173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.113119 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-credential-socket" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:28.118069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.118034 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-certs" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:28.121288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.121085 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-podinfo" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:27:28.121288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.121238 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-kube-api-access-7jtqr" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "kube-api-access-7jtqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:28.121462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.121387 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-envoy" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:28.125397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.125224 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-token" pod "7c8f5d43-c993-449c-a0af-7a250d064b63" (UID: "7c8f5d43-c993-449c-a0af-7a250d064b63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:28.165660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.164925 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.165660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.164960 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtqr\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-kube-api-access-7jtqr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.165660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.164971 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.165660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.164985 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.165660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.164998 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.165660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.165011 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.165660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.165025 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.165660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.165036 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f5d43-c993-449c-a0af-7a250d064b63-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:28.336907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.335850 4431 generic.go:356] "Generic (PLEG): container finished" podID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerID="3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7" exitCode=0 Nov 02 14:27:28.337095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.336930 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" Nov 02 14:27:28.342712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.341042 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" event={"ID":"7c8f5d43-c993-449c-a0af-7a250d064b63","Type":"ContainerDied","Data":"3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7"} Nov 02 14:27:28.342712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.341106 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9" event={"ID":"7c8f5d43-c993-449c-a0af-7a250d064b63","Type":"ContainerDied","Data":"a163c2dd5795e8c6fccf30818adcef31e081d098da2b056350cedd6e920602d3"} Nov 02 14:27:28.342712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.341137 4431 scope.go:122] "RemoveContainer" containerID="3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7" Nov 02 14:27:28.356118 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.356101 4431 scope.go:122] "RemoveContainer" containerID="a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214" Nov 02 14:27:28.368458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.368349 4431 scope.go:122] "RemoveContainer" containerID="3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7" Nov 02 14:27:28.370712 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:28.368882 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7\": container with ID starting with 3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7 not found: ID does not exist" containerID="3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7" Nov 02 14:27:28.370712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.368920 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7"} err="failed to get container status \"3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7\": rpc error: code = NotFound desc = could not find container \"3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7\": container with ID starting with 3292e9a3313a811828f9687e948c2014d030d90ea8b8ca7343c79d1a58ea19f7 not found: ID does not exist" Nov 02 14:27:28.370712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.368945 4431 scope.go:122] "RemoveContainer" containerID="a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214" Nov 02 14:27:28.370712 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:28.369264 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214\": container with ID starting with a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214 not found: ID does not exist" containerID="a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214" Nov 02 14:27:28.370712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.369293 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214"} err="failed to get container status \"a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214\": rpc error: code = NotFound desc = could not find container \"a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214\": container with ID starting with a4c2efe6b5c912d639a04e31c705f15f8a6dcd3110d85ea2e296d31705c10214 not found: ID does not exist" Nov 02 14:27:28.403092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.403057 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9"] Nov 02 14:27:28.422751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:28.422718 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3261bdf2-openshift-default-649b9ff6d9-vk2q9"] Nov 02 14:27:29.076616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:29.076537 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" path="/var/lib/kubelet/pods/7c8f5d43-c993-449c-a0af-7a250d064b63/volumes" Nov 02 14:27:31.267829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.267787 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" podUID="79eb61e4-1bbd-4465-9b5b-85b8e112f12c" containerName="main" containerID="cri-o://2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526" gracePeriod=45 Nov 02 14:27:31.562012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.561984 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:27:31.715158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715131 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-kube-api-access-lf98c\" (UniqueName: \"kubernetes.io/projected/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-kube-api-access-lf98c\") pod \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " Nov 02 14:27:31.715343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715168 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-model-cache\") pod \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " Nov 02 14:27:31.715343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715237 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tls-certs\" (UniqueName: \"kubernetes.io/secret/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tls-certs\") pod \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " Nov 02 14:27:31.715343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715266 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-home\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-home\") pod \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " Nov 02 14:27:31.715515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715408 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tmp-dir\") pod \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " Nov 02 14:27:31.715515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715448 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-dshm\") pod \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\" (UID: \"79eb61e4-1bbd-4465-9b5b-85b8e112f12c\") " Nov 02 14:27:31.715636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715532 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-model-cache" pod "79eb61e4-1bbd-4465-9b5b-85b8e112f12c" (UID: "79eb61e4-1bbd-4465-9b5b-85b8e112f12c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:31.715636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715575 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-home" pod "79eb61e4-1bbd-4465-9b5b-85b8e112f12c" (UID: "79eb61e4-1bbd-4465-9b5b-85b8e112f12c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:31.715736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.715687 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tmp-dir" pod "79eb61e4-1bbd-4465-9b5b-85b8e112f12c" (UID: "79eb61e4-1bbd-4465-9b5b-85b8e112f12c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:31.716032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.716006 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:31.716133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.716034 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:31.716133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.716050 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:31.717520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.717490 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tls-certs" pod "79eb61e4-1bbd-4465-9b5b-85b8e112f12c" (UID: "79eb61e4-1bbd-4465-9b5b-85b8e112f12c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:27:31.717520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.717498 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-kube-api-access-lf98c" pod "79eb61e4-1bbd-4465-9b5b-85b8e112f12c" (UID: "79eb61e4-1bbd-4465-9b5b-85b8e112f12c"). InnerVolumeSpecName "kube-api-access-lf98c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:31.717669 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.717593 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-dshm" pod "79eb61e4-1bbd-4465-9b5b-85b8e112f12c" (UID: "79eb61e4-1bbd-4465-9b5b-85b8e112f12c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:31.817028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.816964 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:31.817028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.816988 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf98c\" (UniqueName: \"kubernetes.io/projected/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-kube-api-access-lf98c\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:31.817028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:31.816999 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79eb61e4-1bbd-4465-9b5b-85b8e112f12c-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:32.378260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.378219 4431 generic.go:356] "Generic (PLEG): container finished" podID="79eb61e4-1bbd-4465-9b5b-85b8e112f12c" containerID="2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526" exitCode=0 Nov 02 14:27:32.378811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.378336 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" Nov 02 14:27:32.378811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.378338 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" event={"ID":"79eb61e4-1bbd-4465-9b5b-85b8e112f12c","Type":"ContainerDied","Data":"2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526"} Nov 02 14:27:32.378811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.378398 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx" event={"ID":"79eb61e4-1bbd-4465-9b5b-85b8e112f12c","Type":"ContainerDied","Data":"4c1cfb88879a93d6bac963dafec301c0cd9618e3a8aff12f1e93729867b5353b"} Nov 02 14:27:32.378811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.378431 4431 scope.go:122] "RemoveContainer" containerID="2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526" Nov 02 14:27:32.391361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.391343 4431 scope.go:122] "RemoveContainer" containerID="2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526" Nov 02 14:27:32.391647 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:32.391629 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526\": container with ID starting with 2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526 not found: ID does not exist" containerID="2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526" Nov 02 14:27:32.391695 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.391656 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526"} err="failed to get container status \"2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526\": rpc error: code = NotFound desc = could not find container \"2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526\": container with ID starting with 2dad1f66e104071a910cdfcdd55bd67e6f61d5cb0558bab78626555c3b43f526 not found: ID does not exist" Nov 02 14:27:32.438636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.438587 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx"] Nov 02 14:27:32.483650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.479900 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-147981/autores-model-147981-kserve-5fdd77f59c-s57cx"] Nov 02 14:27:32.864565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.864534 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4"] Nov 02 14:27:32.865122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.865108 4431 cpu_manager.go:414] "containerMap: removing container" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" Nov 02 14:27:32.865188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.865123 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" Nov 02 14:27:32.865188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.865146 4431 cpu_manager.go:414] "containerMap: removing container" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" Nov 02 14:27:32.865188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.865151 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" Nov 02 14:27:32.865188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.865166 4431 cpu_manager.go:414] "containerMap: removing container" podUID="79eb61e4-1bbd-4465-9b5b-85b8e112f12c" containerName="main" Nov 02 14:27:32.865188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.865171 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79eb61e4-1bbd-4465-9b5b-85b8e112f12c" containerName="main" Nov 02 14:27:32.865344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.865244 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" Nov 02 14:27:32.865344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.865255 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="79eb61e4-1bbd-4465-9b5b-85b8e112f12c" containerName="main" Nov 02 14:27:32.868190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.868172 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:32.886185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.886162 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-d909004fa776\"" Nov 02 14:27:32.889731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.889706 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4"] Nov 02 14:27:32.931251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.931222 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:32.931396 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.931348 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:32.931396 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.931384 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghl55\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-kube-api-access-ghl55\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:32.931474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:32.931461 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.032167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.032136 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.032335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.032230 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.032335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.032268 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghl55\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-kube-api-access-ghl55\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.032335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.032315 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.033027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.033004 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.033123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.033043 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.034987 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.034964 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.056999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.056968 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghl55\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-kube-api-access-ghl55\") pod \"payload-processing-external-model-e2e-extmod-a-944506-649chdmp4\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.077250 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.077221 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79eb61e4-1bbd-4465-9b5b-85b8e112f12c" path="/var/lib/kubelet/pods/79eb61e4-1bbd-4465-9b5b-85b8e112f12c/volumes" Nov 02 14:27:33.180736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.180705 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:33.489147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.489074 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4"] Nov 02 14:27:33.491104 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:27:33.491071 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff16766_b8c1_4bab_83c5_a4732026f528.slice/crio-7192a143f7cd1f837800f6009a7cd2cb82773d9af0be2fbb7edbb3473cb4355f WatchSource:0}: Error finding container 7192a143f7cd1f837800f6009a7cd2cb82773d9af0be2fbb7edbb3473cb4355f: Status 404 returned error can't find the container with id 7192a143f7cd1f837800f6009a7cd2cb82773d9af0be2fbb7edbb3473cb4355f Nov 02 14:27:33.852273 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.852200 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr"] Nov 02 14:27:33.852917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.852902 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8f5d43-c993-449c-a0af-7a250d064b63" containerName="istio-proxy" Nov 02 14:27:33.855210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.855195 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:33.876823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.876798 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr"] Nov 02 14:27:33.879063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.879041 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:27:33.886557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.886501 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:27:33.944075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.944050 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:33.944179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.944109 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjqtm\" (UniqueName: \"kubernetes.io/projected/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-kube-api-access-kjqtm\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:33.944295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:33.944276 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:34.045205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.045169 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:34.045501 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.045325 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjqtm\" (UniqueName: \"kubernetes.io/projected/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-kube-api-access-kjqtm\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:34.045582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.045499 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:34.046350 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.046320 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:34.047954 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.047922 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:34.070270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.070243 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjqtm\" (UniqueName: \"kubernetes.io/projected/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-kube-api-access-kjqtm\") pod \"maas-api-revoke-sub-17cc922d-ssrkr\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:34.167285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.167260 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:34.393298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.393252 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" event={"ID":"0ff16766-b8c1-4bab-83c5-a4732026f528","Type":"ContainerStarted","Data":"934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3"} Nov 02 14:27:34.393464 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.393312 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" event={"ID":"0ff16766-b8c1-4bab-83c5-a4732026f528","Type":"ContainerStarted","Data":"7192a143f7cd1f837800f6009a7cd2cb82773d9af0be2fbb7edbb3473cb4355f"} Nov 02 14:27:34.393464 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.393338 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:34.440381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.440298 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" podStartSLOduration=2.4402841459999998 podStartE2EDuration="2.440284146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:34.438811793 +0000 UTC m=+1819.475916722" watchObservedRunningTime="2026-11-02 14:27:34.440284146 +0000 UTC m=+1819.477389078" Nov 02 14:27:34.480598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:34.480572 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr"] Nov 02 14:27:34.481824 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:27:34.481800 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657b4bbb_d59f_4d70_b6df_d3e3a43d9b2b.slice/crio-5331adad4d1f6aab86232b9209f542a0b34b3257d15c9a8d5195f3f5d38b8704 WatchSource:0}: Error finding container 5331adad4d1f6aab86232b9209f542a0b34b3257d15c9a8d5195f3f5d38b8704: Status 404 returned error can't find the container with id 5331adad4d1f6aab86232b9209f542a0b34b3257d15c9a8d5195f3f5d38b8704 Nov 02 14:27:35.400934 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:35.400902 4431 generic.go:356] "Generic (PLEG): container finished" podID="657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b" containerID="996c62e4d26cc1b227e4fc209af5db04081d56579379426c328e0aad714bc364" exitCode=0 Nov 02 14:27:35.401383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:35.400998 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" event={"ID":"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b","Type":"ContainerDied","Data":"996c62e4d26cc1b227e4fc209af5db04081d56579379426c328e0aad714bc364"} Nov 02 14:27:35.401383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:35.401063 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" event={"ID":"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b","Type":"ContainerStarted","Data":"5331adad4d1f6aab86232b9209f542a0b34b3257d15c9a8d5195f3f5d38b8704"} Nov 02 14:27:36.566915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.566891 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:36.677600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.677521 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-openshift-ca\") pod \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " Nov 02 14:27:36.677765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.677613 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-xks-ca\") pod \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " Nov 02 14:27:36.677860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.677790 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-kube-api-access-kjqtm\" (UniqueName: \"kubernetes.io/projected/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-kube-api-access-kjqtm\") pod \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\" (UID: \"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b\") " Nov 02 14:27:36.677918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.677849 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-openshift-ca" pod "657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b" (UID: "657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:36.678229 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.678211 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:36.679596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.679572 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-xks-ca" pod "657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b" (UID: "657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:27:36.679712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.679693 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-kube-api-access-kjqtm" pod "657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b" (UID: "657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b"). InnerVolumeSpecName "kube-api-access-kjqtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:36.779570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.779549 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:36.779570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:36.779570 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjqtm\" (UniqueName: \"kubernetes.io/projected/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b-kube-api-access-kjqtm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:37.414571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:37.414545 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" Nov 02 14:27:37.414839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:37.414587 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr" event={"ID":"657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b","Type":"ContainerDied","Data":"5331adad4d1f6aab86232b9209f542a0b34b3257d15c9a8d5195f3f5d38b8704"} Nov 02 14:27:37.414839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:37.414634 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5331adad4d1f6aab86232b9209f542a0b34b3257d15c9a8d5195f3f5d38b8704" Nov 02 14:27:41.571904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:41.571870 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:27:44.785123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.785095 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq"] Nov 02 14:27:44.785723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.785708 4431 cpu_manager.go:414] "containerMap: removing container" podUID="657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b" containerName="revoke-keys" Nov 02 14:27:44.785765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.785724 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b" containerName="revoke-keys" Nov 02 14:27:44.785839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.785829 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b" containerName="revoke-keys" Nov 02 14:27:44.788273 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.788258 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:44.818908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.818882 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:27:44.820978 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.820955 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:27:44.822435 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.822414 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq"] Nov 02 14:27:44.881605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.881578 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdr5\" (UniqueName: \"kubernetes.io/projected/ec36bd32-b429-4afc-91e2-8e07250e04e5-kube-api-access-2tdr5\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:44.881724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.881704 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:44.881823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.881808 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:44.983074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.983032 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:44.983680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.983640 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:44.983874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.983809 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdr5\" (UniqueName: \"kubernetes.io/projected/ec36bd32-b429-4afc-91e2-8e07250e04e5-kube-api-access-2tdr5\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:44.984882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.984857 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:44.987528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:44.987502 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:45.009135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:45.009102 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdr5\" (UniqueName: \"kubernetes.io/projected/ec36bd32-b429-4afc-91e2-8e07250e04e5-kube-api-access-2tdr5\") pod \"maas-api-revoke-sub-52e36c10-kx9mq\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:45.101091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:45.101022 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:45.456906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:45.455999 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:27:45.474855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:45.471106 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq"] Nov 02 14:27:45.557566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:45.557538 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7"] Nov 02 14:27:46.541737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:46.541696 4431 generic.go:356] "Generic (PLEG): container finished" podID="ec36bd32-b429-4afc-91e2-8e07250e04e5" containerID="b7fadf0aba84d0d8b3229680053270ec5d44bb1d8ae2ec038f8e1f8787e54889" exitCode=0 Nov 02 14:27:46.542118 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:46.541793 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" event={"ID":"ec36bd32-b429-4afc-91e2-8e07250e04e5","Type":"ContainerDied","Data":"b7fadf0aba84d0d8b3229680053270ec5d44bb1d8ae2ec038f8e1f8787e54889"} Nov 02 14:27:46.542118 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:46.541847 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" event={"ID":"ec36bd32-b429-4afc-91e2-8e07250e04e5","Type":"ContainerStarted","Data":"fc880d8f02f87a5730ee94ceb257fa9597fbe230da681dfc8610ac3271e9a14f"} Nov 02 14:27:47.696166 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.696132 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:47.756155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.756126 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec36bd32-b429-4afc-91e2-8e07250e04e5-kube-api-access-2tdr5\" (UniqueName: \"kubernetes.io/projected/ec36bd32-b429-4afc-91e2-8e07250e04e5-kube-api-access-2tdr5\") pod \"ec36bd32-b429-4afc-91e2-8e07250e04e5\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " Nov 02 14:27:47.756319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.756222 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-openshift-ca\") pod \"ec36bd32-b429-4afc-91e2-8e07250e04e5\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " Nov 02 14:27:47.756319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.756293 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-xks-ca\") pod \"ec36bd32-b429-4afc-91e2-8e07250e04e5\" (UID: \"ec36bd32-b429-4afc-91e2-8e07250e04e5\") " Nov 02 14:27:47.756649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.756619 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-openshift-ca" pod "ec36bd32-b429-4afc-91e2-8e07250e04e5" (UID: "ec36bd32-b429-4afc-91e2-8e07250e04e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:47.756859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.756839 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:47.758433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.758403 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-xks-ca" pod "ec36bd32-b429-4afc-91e2-8e07250e04e5" (UID: "ec36bd32-b429-4afc-91e2-8e07250e04e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:27:47.758556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.758449 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec36bd32-b429-4afc-91e2-8e07250e04e5-kube-api-access-2tdr5" pod "ec36bd32-b429-4afc-91e2-8e07250e04e5" (UID: "ec36bd32-b429-4afc-91e2-8e07250e04e5"). InnerVolumeSpecName "kube-api-access-2tdr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:47.858108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.858044 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec36bd32-b429-4afc-91e2-8e07250e04e5-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:47.858108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:47.858071 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tdr5\" (UniqueName: \"kubernetes.io/projected/ec36bd32-b429-4afc-91e2-8e07250e04e5-kube-api-access-2tdr5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:48.557066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:48.557038 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" Nov 02 14:27:48.557275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:48.557043 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq" event={"ID":"ec36bd32-b429-4afc-91e2-8e07250e04e5","Type":"ContainerDied","Data":"fc880d8f02f87a5730ee94ceb257fa9597fbe230da681dfc8610ac3271e9a14f"} Nov 02 14:27:48.557275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:48.557148 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc880d8f02f87a5730ee94ceb257fa9597fbe230da681dfc8610ac3271e9a14f" Nov 02 14:27:49.252419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:49.252384 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz"] Nov 02 14:27:49.252975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:49.252653 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" containerID="cri-o://73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33" gracePeriod=30 Nov 02 14:27:50.570441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.569839 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" podUID="cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" containerName="payload-processing" containerID="cri-o://29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec" gracePeriod=25 Nov 02 14:27:50.869753 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.869714 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:27:50.895238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.895209 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-provider-credentials\") pod \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " Nov 02 14:27:50.895448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.895258 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-kube-api-access-j8ftt\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-kube-api-access-j8ftt\") pod \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " Nov 02 14:27:50.895448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.895377 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-routing-overlay-volume\") pod \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " Nov 02 14:27:50.895597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.895478 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-plugins-config-volume\") pod \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\" (UID: \"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e\") " Nov 02 14:27:50.895831 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.895773 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-routing-overlay-volume" pod "cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" (UID: "cc9eafd5-2b84-4dfb-8b07-d8be5e88857e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:50.895943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.895865 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-plugins-config-volume" pod "cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" (UID: "cc9eafd5-2b84-4dfb-8b07-d8be5e88857e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:50.896071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.896051 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:50.896071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.896072 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:50.898051 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.898027 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-provider-credentials" pod "cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" (UID: "cc9eafd5-2b84-4dfb-8b07-d8be5e88857e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:50.898133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.898057 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-kube-api-access-j8ftt" pod "cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" (UID: "cc9eafd5-2b84-4dfb-8b07-d8be5e88857e"). InnerVolumeSpecName "kube-api-access-j8ftt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:50.997878 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.997846 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:50.997878 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:50.997874 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ftt\" (UniqueName: \"kubernetes.io/projected/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e-kube-api-access-j8ftt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:51.286985 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:51.286947 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:27:51.587061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.587031 4431 generic.go:356] "Generic (PLEG): container finished" podID="cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" containerID="29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec" exitCode=0 Nov 02 14:27:51.587486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.587081 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" Nov 02 14:27:51.587486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.587109 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" event={"ID":"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e","Type":"ContainerDied","Data":"29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec"} Nov 02 14:27:51.587486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.587157 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7" event={"ID":"cc9eafd5-2b84-4dfb-8b07-d8be5e88857e","Type":"ContainerDied","Data":"049ea2c73e03254edd3b4e2cac88594670acdc119e018a14bab1bc3eeead30b7"} Nov 02 14:27:51.587486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.587179 4431 scope.go:122] "RemoveContainer" containerID="29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec" Nov 02 14:27:51.596603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.596587 4431 scope.go:122] "RemoveContainer" containerID="29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec" Nov 02 14:27:51.596886 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:51.596869 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec\": container with ID starting with 29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec not found: ID does not exist" containerID="29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec" Nov 02 14:27:51.596961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.596905 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec"} err="failed to get container status \"29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec\": rpc error: code = NotFound desc = could not find container \"29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec\": container with ID starting with 29382d1e3d9c802af027e260c70a468168e1372559bde7edcf893cb7b9479bec not found: ID does not exist" Nov 02 14:27:51.646265 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.646245 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7"] Nov 02 14:27:51.662535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:51.662511 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-5f69n9vv7"] Nov 02 14:27:53.076215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:53.076183 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" path="/var/lib/kubelet/pods/cc9eafd5-2b84-4dfb-8b07-d8be5e88857e/volumes" Nov 02 14:27:54.586653 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.586628 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:27:54.607309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.607281 4431 generic.go:356] "Generic (PLEG): container finished" podID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerID="73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33" exitCode=0 Nov 02 14:27:54.607436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.607326 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" Nov 02 14:27:54.607436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.607344 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" event={"ID":"6be03ff8-cbed-47db-8ad1-b071adfb2865","Type":"ContainerDied","Data":"73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33"} Nov 02 14:27:54.607436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.607398 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz" event={"ID":"6be03ff8-cbed-47db-8ad1-b071adfb2865","Type":"ContainerDied","Data":"f1307a8943cddd50ad735438731cc243a81b85372bb0421f8d58afe24ea25582"} Nov 02 14:27:54.607436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.607429 4431 scope.go:122] "RemoveContainer" containerID="73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33" Nov 02 14:27:54.617415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.617397 4431 scope.go:122] "RemoveContainer" containerID="9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41" Nov 02 14:27:54.626248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.626230 4431 scope.go:122] "RemoveContainer" containerID="73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33" Nov 02 14:27:54.626536 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:54.626511 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33\": container with ID starting with 73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33 not found: ID does not exist" containerID="73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33" Nov 02 14:27:54.626640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.626540 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33"} err="failed to get container status \"73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33\": rpc error: code = NotFound desc = could not find container \"73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33\": container with ID starting with 73403e20fd0d04e6cfd48557c9d37ad1ca4dc218cf587f049d7a2706815e8f33 not found: ID does not exist" Nov 02 14:27:54.626640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.626557 4431 scope.go:122] "RemoveContainer" containerID="9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41" Nov 02 14:27:54.626845 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:54.626828 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41\": container with ID starting with 9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41 not found: ID does not exist" containerID="9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41" Nov 02 14:27:54.626897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.626851 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41"} err="failed to get container status \"9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41\": rpc error: code = NotFound desc = could not find container \"9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41\": container with ID starting with 9487ba9e67c222b2434d6b7e50828cf8225cc555756e8ace4e0b343458d7af41 not found: ID does not exist" Nov 02 14:27:54.643621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.643592 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-podinfo\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.643736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.643654 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-socket\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.643736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.643724 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-kube-api-access-zm58v\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-kube-api-access-zm58v\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.643901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.643761 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be03ff8-cbed-47db-8ad1-b071adfb2865-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be03ff8-cbed-47db-8ad1-b071adfb2865-istiod-ca-cert\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.643901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.643843 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-credential-socket\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.644015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.643920 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-token\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-token\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.644015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.643950 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-certs\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.644015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.643980 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-socket" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:54.644171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644025 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-envoy\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.644171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644134 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-credential-socket" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:54.644171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644153 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-data\") pod \"6be03ff8-cbed-47db-8ad1-b071adfb2865\" (UID: \"6be03ff8-cbed-47db-8ad1-b071adfb2865\") " Nov 02 14:27:54.644316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644231 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be03ff8-cbed-47db-8ad1-b071adfb2865-istiod-ca-cert" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:27:54.644368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644312 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-certs" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:54.644593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644560 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-data" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:54.644719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644693 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.644719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644714 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.644871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644727 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6be03ff8-cbed-47db-8ad1-b071adfb2865-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.644871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644741 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.644871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.644754 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.647830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.647245 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-kube-api-access-zm58v" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "kube-api-access-zm58v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:54.647830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.647272 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-envoy" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:27:54.647830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.647312 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-podinfo" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:27:54.647830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.647350 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-token" pod "6be03ff8-cbed-47db-8ad1-b071adfb2865" (UID: "6be03ff8-cbed-47db-8ad1-b071adfb2865"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:54.746052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.746022 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.746052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.746056 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.746052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.746066 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm58v\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-kube-api-access-zm58v\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.746052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.746076 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6be03ff8-cbed-47db-8ad1-b071adfb2865-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:54.965722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.965690 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz"] Nov 02 14:27:54.968391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:54.968369 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-147981-openshift-default-76d9954467-94hjz"] Nov 02 14:27:55.077533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:55.077504 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" path="/var/lib/kubelet/pods/6be03ff8-cbed-47db-8ad1-b071adfb2865/volumes" Nov 02 14:27:55.767625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:55.767592 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq"] Nov 02 14:27:55.768090 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:55.767795 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" containerID="cri-o://41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318" gracePeriod=30 Nov 02 14:27:57.246336 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.246301 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx"] Nov 02 14:27:57.246727 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.246493 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" podUID="6a0010ea-8eef-4b5f-b1fc-b85d49aa5607" containerName="sim" containerID="cri-o://6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe" gracePeriod=30 Nov 02 14:27:57.631821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.628548 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:27:57.639128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.639096 4431 generic.go:356] "Generic (PLEG): container finished" podID="6a0010ea-8eef-4b5f-b1fc-b85d49aa5607" containerID="6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe" exitCode=0 Nov 02 14:27:57.639370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.639219 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" event={"ID":"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607","Type":"ContainerDied","Data":"6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe"} Nov 02 14:27:57.639370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.639250 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" event={"ID":"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607","Type":"ContainerDied","Data":"f8eb112529a91498f69fe93d02592fdaf4a7a6d49120660408bc510e692132da"} Nov 02 14:27:57.639370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.639267 4431 scope.go:122] "RemoveContainer" containerID="6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe" Nov 02 14:27:57.639645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.639389 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx" Nov 02 14:27:57.650145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.649845 4431 scope.go:122] "RemoveContainer" containerID="6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe" Nov 02 14:27:57.650270 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:27:57.650157 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe\": container with ID starting with 6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe not found: ID does not exist" containerID="6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe" Nov 02 14:27:57.650270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.650192 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe"} err="failed to get container status \"6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe\": rpc error: code = NotFound desc = could not find container \"6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe\": container with ID starting with 6eb525905af26201e656e3296bcd31b39a83d7128476b20edc801669ed0f0dfe not found: ID does not exist" Nov 02 14:27:57.702038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.702009 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-kube-api-access-xvvps\" (UniqueName: \"kubernetes.io/projected/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-kube-api-access-xvvps\") pod \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\" (UID: \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\") " Nov 02 14:27:57.702178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.702049 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-tls\" (UniqueName: \"kubernetes.io/secret/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-tls\") pod \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\" (UID: \"6a0010ea-8eef-4b5f-b1fc-b85d49aa5607\") " Nov 02 14:27:57.704959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.704929 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-tls" pod "6a0010ea-8eef-4b5f-b1fc-b85d49aa5607" (UID: "6a0010ea-8eef-4b5f-b1fc-b85d49aa5607"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:27:57.705118 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.705098 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-kube-api-access-xvvps" pod "6a0010ea-8eef-4b5f-b1fc-b85d49aa5607" (UID: "6a0010ea-8eef-4b5f-b1fc-b85d49aa5607"). InnerVolumeSpecName "kube-api-access-xvvps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:27:57.804175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.804087 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvps\" (UniqueName: \"kubernetes.io/projected/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-kube-api-access-xvvps\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:57.804175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:57.804111 4431 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:27:58.002533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:58.002505 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx"] Nov 02 14:27:58.006758 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:58.006731 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-c05168/e2e-usage-sim-c8b392-54bf74765c-5dhbx"] Nov 02 14:27:59.076190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:27:59.076159 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0010ea-8eef-4b5f-b1fc-b85d49aa5607" path="/var/lib/kubelet/pods/6a0010ea-8eef-4b5f-b1fc-b85d49aa5607/volumes" Nov 02 14:28:00.422587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.422551 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp"] Nov 02 14:28:00.423302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423281 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ec36bd32-b429-4afc-91e2-8e07250e04e5" containerName="revoke-keys" Nov 02 14:28:00.423409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423303 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec36bd32-b429-4afc-91e2-8e07250e04e5" containerName="revoke-keys" Nov 02 14:28:00.423409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423328 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" Nov 02 14:28:00.423409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423336 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" Nov 02 14:28:00.423409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423383 4431 cpu_manager.go:414] "containerMap: removing container" podUID="cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" containerName="payload-processing" Nov 02 14:28:00.423409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423391 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" containerName="payload-processing" Nov 02 14:28:00.423661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423416 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6a0010ea-8eef-4b5f-b1fc-b85d49aa5607" containerName="sim" Nov 02 14:28:00.423661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423424 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0010ea-8eef-4b5f-b1fc-b85d49aa5607" containerName="sim" Nov 02 14:28:00.423661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423551 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9eafd5-2b84-4dfb-8b07-d8be5e88857e" containerName="payload-processing" Nov 02 14:28:00.423661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423564 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" Nov 02 14:28:00.423661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423577 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0010ea-8eef-4b5f-b1fc-b85d49aa5607" containerName="sim" Nov 02 14:28:00.423661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.423588 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec36bd32-b429-4afc-91e2-8e07250e04e5" containerName="revoke-keys" Nov 02 14:28:00.426584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.426564 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.457037 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.457015 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:28:00.457177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.457108 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:28:00.462131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.462105 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp"] Nov 02 14:28:00.558952 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.558916 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.558952 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.558952 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkgn\" (UniqueName: \"kubernetes.io/projected/eab3d2c5-e98a-467a-9c8d-945263033493-kube-api-access-9pkgn\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.559136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.559088 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.660345 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.660316 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.660506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.660434 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.660506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.660457 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkgn\" (UniqueName: \"kubernetes.io/projected/eab3d2c5-e98a-467a-9c8d-945263033493-kube-api-access-9pkgn\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.661102 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.661076 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.662859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.662838 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.692272 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.692210 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkgn\" (UniqueName: \"kubernetes.io/projected/eab3d2c5-e98a-467a-9c8d-945263033493-kube-api-access-9pkgn\") pod \"maas-api-revoke-sub-1b2e85a0-7nzhp\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:00.738763 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:00.738735 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:01.015863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.015840 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:28:01.048044 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.047995 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp"] Nov 02 14:28:01.048169 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:01.048077 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab3d2c5_e98a_467a_9c8d_945263033493.slice/crio-24a437c518c64ea9d56b2336cf0282a96f76524504e6e6e80f8ab2a83f140651 WatchSource:0}: Error finding container 24a437c518c64ea9d56b2336cf0282a96f76524504e6e6e80f8ab2a83f140651: Status 404 returned error can't find the container with id 24a437c518c64ea9d56b2336cf0282a96f76524504e6e6e80f8ab2a83f140651 Nov 02 14:28:01.165671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.165643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-socket\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.165864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.165700 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-kube-api-access-7w7mt\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-kube-api-access-7w7mt\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.165864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.165761 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-certs\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.165990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.165867 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-envoy\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.165990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.165893 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-credential-socket\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.165990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.165934 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3afbe79c-479d-46f5-9d36-d1443461695f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afbe79c-479d-46f5-9d36-d1443461695f-istio-podinfo\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.165990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.165952 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-istio-token\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-istio-token\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.165990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.165965 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-socket" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.166264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166009 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afbe79c-479d-46f5-9d36-d1443461695f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afbe79c-479d-46f5-9d36-d1443461695f-istiod-ca-cert\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.166264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166119 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-certs" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.166264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166198 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-data\") pod \"3afbe79c-479d-46f5-9d36-d1443461695f\" (UID: \"3afbe79c-479d-46f5-9d36-d1443461695f\") " Nov 02 14:28:01.166264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166218 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-credential-socket" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.166470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166452 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-data" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.166546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166525 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afbe79c-479d-46f5-9d36-d1443461695f-istiod-ca-cert" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:01.166816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166761 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.166936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166818 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.166936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166836 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.166936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166849 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.166936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.166862 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afbe79c-479d-46f5-9d36-d1443461695f-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.168214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.168186 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3afbe79c-479d-46f5-9d36-d1443461695f-istio-podinfo" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:28:01.168321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.168243 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-kube-api-access-7w7mt" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "kube-api-access-7w7mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:01.168547 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.168520 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-envoy" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:01.168614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.168527 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-istio-token" pod "3afbe79c-479d-46f5-9d36-d1443461695f" (UID: "3afbe79c-479d-46f5-9d36-d1443461695f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:01.268390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.268309 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afbe79c-479d-46f5-9d36-d1443461695f-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.268390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.268339 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afbe79c-479d-46f5-9d36-d1443461695f-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.268390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.268351 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.268390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.268362 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w7mt\" (UniqueName: \"kubernetes.io/projected/3afbe79c-479d-46f5-9d36-d1443461695f-kube-api-access-7w7mt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:01.271173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.271146 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc"] Nov 02 14:28:01.271773 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.271753 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" Nov 02 14:28:01.271773 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.271771 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" Nov 02 14:28:01.271933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.271805 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" Nov 02 14:28:01.271933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.271810 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" Nov 02 14:28:01.271933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.271914 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be03ff8-cbed-47db-8ad1-b071adfb2865" containerName="istio-proxy" Nov 02 14:28:01.271933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.271926 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" containerName="istio-proxy" Nov 02 14:28:01.275093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.275069 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.295358 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.295339 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc"] Nov 02 14:28:01.369568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.369537 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.369733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.369600 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.369733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.369674 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.369733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.369722 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsghm\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-kube-api-access-bsghm\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.471171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.471142 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.471631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.471186 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.471631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.471229 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsghm\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-kube-api-access-bsghm\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.471631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.471340 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.472011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.471991 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.472237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.472214 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.473686 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.473663 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.496751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.496727 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsghm\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-kube-api-access-bsghm\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.598405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.598321 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:01.673165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.673131 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" event={"ID":"eab3d2c5-e98a-467a-9c8d-945263033493","Type":"ContainerStarted","Data":"f8f7a3557b1ba3dac573b8ad380398ad46407b9bfdde6a43cad9b3f11e42aa43"} Nov 02 14:28:01.673331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.673193 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" event={"ID":"eab3d2c5-e98a-467a-9c8d-945263033493","Type":"ContainerStarted","Data":"24a437c518c64ea9d56b2336cf0282a96f76524504e6e6e80f8ab2a83f140651"} Nov 02 14:28:01.676257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.676227 4431 generic.go:356] "Generic (PLEG): container finished" podID="3afbe79c-479d-46f5-9d36-d1443461695f" containerID="41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318" exitCode=0 Nov 02 14:28:01.676387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.676283 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" event={"ID":"3afbe79c-479d-46f5-9d36-d1443461695f","Type":"ContainerDied","Data":"41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318"} Nov 02 14:28:01.676387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.676344 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" event={"ID":"3afbe79c-479d-46f5-9d36-d1443461695f","Type":"ContainerDied","Data":"cf81a6518e5ef8cc33869e15bbd0edabc69e52486df28a5024a6af33d0eb4eac"} Nov 02 14:28:01.676387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.676359 4431 scope.go:122] "RemoveContainer" containerID="41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318" Nov 02 14:28:01.676651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.676625 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq" Nov 02 14:28:01.703739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.703654 4431 scope.go:122] "RemoveContainer" containerID="41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318" Nov 02 14:28:01.704209 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:01.704175 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318\": container with ID starting with 41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318 not found: ID does not exist" containerID="41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318" Nov 02 14:28:01.704318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.704221 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318"} err="failed to get container status \"41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318\": rpc error: code = NotFound desc = could not find container \"41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318\": container with ID starting with 41efb563be83661f1e7ee2745a51b236097891daaf7f15473dd16662c25ec318 not found: ID does not exist" Nov 02 14:28:01.720048 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.719999 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" podStartSLOduration=1.7199872969999999 podStartE2EDuration="1.719987297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:01.718144806 +0000 UTC m=+1846.755249747" watchObservedRunningTime="2026-11-02 14:28:01.719987297 +0000 UTC m=+1846.757092228" Nov 02 14:28:01.760919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.760889 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq"] Nov 02 14:28:01.765133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.765108 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-lj8bq"] Nov 02 14:28:01.912415 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:01.912387 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aaa6f9_172c_48fe_96ac_b29f4a554364.slice/crio-919b2e8d6b538c1739a1594827224104f4fe7f703c051a24efcece02d5994fc1 WatchSource:0}: Error finding container 919b2e8d6b538c1739a1594827224104f4fe7f703c051a24efcece02d5994fc1: Status 404 returned error can't find the container with id 919b2e8d6b538c1739a1594827224104f4fe7f703c051a24efcece02d5994fc1 Nov 02 14:28:01.940601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:01.940574 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc"] Nov 02 14:28:02.685077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:02.685039 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" event={"ID":"e6aaa6f9-172c-48fe-96ac-b29f4a554364","Type":"ContainerStarted","Data":"06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439"} Nov 02 14:28:02.685565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:02.685108 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" event={"ID":"e6aaa6f9-172c-48fe-96ac-b29f4a554364","Type":"ContainerStarted","Data":"919b2e8d6b538c1739a1594827224104f4fe7f703c051a24efcece02d5994fc1"} Nov 02 14:28:02.685565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:02.685213 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:02.686761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:02.686738 4431 generic.go:356] "Generic (PLEG): container finished" podID="eab3d2c5-e98a-467a-9c8d-945263033493" containerID="f8f7a3557b1ba3dac573b8ad380398ad46407b9bfdde6a43cad9b3f11e42aa43" exitCode=0 Nov 02 14:28:02.686882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:02.686771 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" event={"ID":"eab3d2c5-e98a-467a-9c8d-945263033493","Type":"ContainerDied","Data":"f8f7a3557b1ba3dac573b8ad380398ad46407b9bfdde6a43cad9b3f11e42aa43"} Nov 02 14:28:02.733414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:02.733364 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" podStartSLOduration=1.7333502570000001 podStartE2EDuration="1.733350257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:02.73094698 +0000 UTC m=+1847.768051911" watchObservedRunningTime="2026-11-02 14:28:02.733350257 +0000 UTC m=+1847.770455189" Nov 02 14:28:03.075990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:03.075909 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afbe79c-479d-46f5-9d36-d1443461695f" path="/var/lib/kubelet/pods/3afbe79c-479d-46f5-9d36-d1443461695f/volumes" Nov 02 14:28:03.856178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:03.856154 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:04.007343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.007268 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-openshift-ca\") pod \"eab3d2c5-e98a-467a-9c8d-945263033493\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " Nov 02 14:28:04.007502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.007382 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab3d2c5-e98a-467a-9c8d-945263033493-kube-api-access-9pkgn\" (UniqueName: \"kubernetes.io/projected/eab3d2c5-e98a-467a-9c8d-945263033493-kube-api-access-9pkgn\") pod \"eab3d2c5-e98a-467a-9c8d-945263033493\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " Nov 02 14:28:04.007502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.007484 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-xks-ca\") pod \"eab3d2c5-e98a-467a-9c8d-945263033493\" (UID: \"eab3d2c5-e98a-467a-9c8d-945263033493\") " Nov 02 14:28:04.007646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.007623 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-openshift-ca" pod "eab3d2c5-e98a-467a-9c8d-945263033493" (UID: "eab3d2c5-e98a-467a-9c8d-945263033493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:04.007909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.007890 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:04.009574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.009555 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-xks-ca" pod "eab3d2c5-e98a-467a-9c8d-945263033493" (UID: "eab3d2c5-e98a-467a-9c8d-945263033493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:04.009666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.009555 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab3d2c5-e98a-467a-9c8d-945263033493-kube-api-access-9pkgn" pod "eab3d2c5-e98a-467a-9c8d-945263033493" (UID: "eab3d2c5-e98a-467a-9c8d-945263033493"). InnerVolumeSpecName "kube-api-access-9pkgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:04.109364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.109334 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab3d2c5-e98a-467a-9c8d-945263033493-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:04.109364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.109359 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkgn\" (UniqueName: \"kubernetes.io/projected/eab3d2c5-e98a-467a-9c8d-945263033493-kube-api-access-9pkgn\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:04.701916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.701887 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" Nov 02 14:28:04.701916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.701903 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp" event={"ID":"eab3d2c5-e98a-467a-9c8d-945263033493","Type":"ContainerDied","Data":"24a437c518c64ea9d56b2336cf0282a96f76524504e6e6e80f8ab2a83f140651"} Nov 02 14:28:04.702134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:04.701965 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24a437c518c64ea9d56b2336cf0282a96f76524504e6e6e80f8ab2a83f140651" Nov 02 14:28:11.315107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.315077 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4"] Nov 02 14:28:11.315599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.315586 4431 cpu_manager.go:414] "containerMap: removing container" podUID="eab3d2c5-e98a-467a-9c8d-945263033493" containerName="revoke-keys" Nov 02 14:28:11.315599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.315599 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab3d2c5-e98a-467a-9c8d-945263033493" containerName="revoke-keys" Nov 02 14:28:11.315726 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.315717 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab3d2c5-e98a-467a-9c8d-945263033493" containerName="revoke-keys" Nov 02 14:28:11.318378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.318362 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.337847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.337824 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:28:11.350349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.350327 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:28:11.353581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.353553 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4"] Nov 02 14:28:11.402769 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.402726 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72gr\" (UniqueName: \"kubernetes.io/projected/403f41ee-757a-4e26-836f-e8c454f7d039-kube-api-access-s72gr\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.403253 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.403224 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.403502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.403485 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.504456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.504419 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72gr\" (UniqueName: \"kubernetes.io/projected/403f41ee-757a-4e26-836f-e8c454f7d039-kube-api-access-s72gr\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.504651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.504537 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.504651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.504556 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.505237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.505209 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.507029 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.507004 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.529195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.529164 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72gr\" (UniqueName: \"kubernetes.io/projected/403f41ee-757a-4e26-836f-e8c454f7d039-kube-api-access-s72gr\") pod \"maas-api-revoke-sub-cff2cd52-9pzv4\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.632414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.632338 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:11.960050 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:11.960024 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4"] Nov 02 14:28:12.810272 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:12.810193 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" event={"ID":"403f41ee-757a-4e26-836f-e8c454f7d039","Type":"ContainerStarted","Data":"b296f0e4bdd990d20d76c3191c1d5411fdd12b6e54835218623ba43fdd003c04"} Nov 02 14:28:12.810272 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:12.810247 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" event={"ID":"403f41ee-757a-4e26-836f-e8c454f7d039","Type":"ContainerStarted","Data":"71a5569821d5818048eb690db2d130e895e80de5e85a29a728ab6e054aab3d65"} Nov 02 14:28:12.856959 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:12.856912 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" podStartSLOduration=1.856898809 podStartE2EDuration="1.856898809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:12.853439513 +0000 UTC m=+1857.890544457" watchObservedRunningTime="2026-11-02 14:28:12.856898809 +0000 UTC m=+1857.894003783" Nov 02 14:28:13.701487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:13.701444 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:13.802265 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:13.802112 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr"] Nov 02 14:28:13.824720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:13.824687 4431 generic.go:356] "Generic (PLEG): container finished" podID="403f41ee-757a-4e26-836f-e8c454f7d039" containerID="b296f0e4bdd990d20d76c3191c1d5411fdd12b6e54835218623ba43fdd003c04" exitCode=0 Nov 02 14:28:13.824720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:13.824718 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" event={"ID":"403f41ee-757a-4e26-836f-e8c454f7d039","Type":"ContainerDied","Data":"b296f0e4bdd990d20d76c3191c1d5411fdd12b6e54835218623ba43fdd003c04"} Nov 02 14:28:15.037382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.037358 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:15.151513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.151482 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-xks-ca\") pod \"403f41ee-757a-4e26-836f-e8c454f7d039\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " Nov 02 14:28:15.151689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.151671 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-openshift-ca\") pod \"403f41ee-757a-4e26-836f-e8c454f7d039\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " Nov 02 14:28:15.151790 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.151726 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403f41ee-757a-4e26-836f-e8c454f7d039-kube-api-access-s72gr\" (UniqueName: \"kubernetes.io/projected/403f41ee-757a-4e26-836f-e8c454f7d039-kube-api-access-s72gr\") pod \"403f41ee-757a-4e26-836f-e8c454f7d039\" (UID: \"403f41ee-757a-4e26-836f-e8c454f7d039\") " Nov 02 14:28:15.152097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.152071 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-openshift-ca" pod "403f41ee-757a-4e26-836f-e8c454f7d039" (UID: "403f41ee-757a-4e26-836f-e8c454f7d039"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:15.153538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.153510 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-xks-ca" pod "403f41ee-757a-4e26-836f-e8c454f7d039" (UID: "403f41ee-757a-4e26-836f-e8c454f7d039"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:15.153708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.153690 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403f41ee-757a-4e26-836f-e8c454f7d039-kube-api-access-s72gr" pod "403f41ee-757a-4e26-836f-e8c454f7d039" (UID: "403f41ee-757a-4e26-836f-e8c454f7d039"). InnerVolumeSpecName "kube-api-access-s72gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:15.253610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.253581 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:15.253610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.253606 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s72gr\" (UniqueName: \"kubernetes.io/projected/403f41ee-757a-4e26-836f-e8c454f7d039-kube-api-access-s72gr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:15.253610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.253616 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403f41ee-757a-4e26-836f-e8c454f7d039-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:15.839883 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.839853 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" event={"ID":"403f41ee-757a-4e26-836f-e8c454f7d039","Type":"ContainerDied","Data":"71a5569821d5818048eb690db2d130e895e80de5e85a29a728ab6e054aab3d65"} Nov 02 14:28:15.840068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.839892 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4" Nov 02 14:28:15.840068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:15.839902 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a5569821d5818048eb690db2d130e895e80de5e85a29a728ab6e054aab3d65" Nov 02 14:28:17.195926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:17.195897 4431 scope.go:122] "RemoveContainer" containerID="05d9487656395b36d88b1d5d009d9094e80eab5fc0181dc0645d9d08973fc193" Nov 02 14:28:17.207801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:17.207769 4431 scope.go:122] "RemoveContainer" containerID="b6854044271b1d79379a74cb8eda5ab4375d80fc4c09082b6d71934e8b31c4d0" Nov 02 14:28:17.216545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:17.216518 4431 scope.go:122] "RemoveContainer" containerID="ad4c1ee70bd9e61e0d615b52b8eb0932da2fc2a48c9a6166a1ff08e0dd72bab6" Nov 02 14:28:17.226015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:17.225997 4431 scope.go:122] "RemoveContainer" containerID="4fd1f899050d0eec9a8b400fc26864b872728c34877ccbfd3c09429051aafb62" Nov 02 14:28:18.802547 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:18.802442 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" podUID="a947cb83-0905-462f-8b99-dc2da8238f81" containerName="payload-processing" containerID="cri-o://1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b" gracePeriod=25 Nov 02 14:28:19.079003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.078979 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:28:19.082644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.082613 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf"] Nov 02 14:28:19.083153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.083135 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a947cb83-0905-462f-8b99-dc2da8238f81" containerName="payload-processing" Nov 02 14:28:19.083153 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.083152 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a947cb83-0905-462f-8b99-dc2da8238f81" containerName="payload-processing" Nov 02 14:28:19.083298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.083167 4431 cpu_manager.go:414] "containerMap: removing container" podUID="403f41ee-757a-4e26-836f-e8c454f7d039" containerName="revoke-keys" Nov 02 14:28:19.083298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.083172 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403f41ee-757a-4e26-836f-e8c454f7d039" containerName="revoke-keys" Nov 02 14:28:19.083298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.083265 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a947cb83-0905-462f-8b99-dc2da8238f81" containerName="payload-processing" Nov 02 14:28:19.083298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.083274 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="403f41ee-757a-4e26-836f-e8c454f7d039" containerName="revoke-keys" Nov 02 14:28:19.086405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.086386 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.107101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.107079 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:28:19.107101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.107089 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:28:19.110735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.110714 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf"] Nov 02 14:28:19.212026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.211993 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-routing-overlay-volume\") pod \"a947cb83-0905-462f-8b99-dc2da8238f81\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " Nov 02 14:28:19.212222 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.212198 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-provider-credentials\") pod \"a947cb83-0905-462f-8b99-dc2da8238f81\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " Nov 02 14:28:19.212297 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.212232 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-plugins-config-volume\") pod \"a947cb83-0905-462f-8b99-dc2da8238f81\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " Nov 02 14:28:19.212362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.212300 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-kube-api-access-lmtft\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-kube-api-access-lmtft\") pod \"a947cb83-0905-462f-8b99-dc2da8238f81\" (UID: \"a947cb83-0905-462f-8b99-dc2da8238f81\") " Nov 02 14:28:19.212445 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.212418 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-routing-overlay-volume" pod "a947cb83-0905-462f-8b99-dc2da8238f81" (UID: "a947cb83-0905-462f-8b99-dc2da8238f81"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:19.212641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.212612 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.212752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.212692 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.212752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.212710 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-plugins-config-volume" pod "a947cb83-0905-462f-8b99-dc2da8238f81" (UID: "a947cb83-0905-462f-8b99-dc2da8238f81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:19.212926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.212905 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpp7z\" (UniqueName: \"kubernetes.io/projected/4a665063-5a9e-4f57-add5-817979f6f382-kube-api-access-bpp7z\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.213244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.213224 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:19.213341 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.213249 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a947cb83-0905-462f-8b99-dc2da8238f81-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:19.214426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.214406 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-provider-credentials" pod "a947cb83-0905-462f-8b99-dc2da8238f81" (UID: "a947cb83-0905-462f-8b99-dc2da8238f81"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:19.214509 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.214479 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-kube-api-access-lmtft" pod "a947cb83-0905-462f-8b99-dc2da8238f81" (UID: "a947cb83-0905-462f-8b99-dc2da8238f81"). InnerVolumeSpecName "kube-api-access-lmtft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:19.315160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.315066 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.315160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.315115 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.315416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.315222 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpp7z\" (UniqueName: \"kubernetes.io/projected/4a665063-5a9e-4f57-add5-817979f6f382-kube-api-access-bpp7z\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.315416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.315298 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:19.315416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.315314 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmtft\" (UniqueName: \"kubernetes.io/projected/a947cb83-0905-462f-8b99-dc2da8238f81-kube-api-access-lmtft\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:19.315826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.315797 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.317475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.317456 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.340274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.340246 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpp7z\" (UniqueName: \"kubernetes.io/projected/4a665063-5a9e-4f57-add5-817979f6f382-kube-api-access-bpp7z\") pod \"maas-api-revoke-sub-c86c866c-fs9nf\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.400956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.400927 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:19.726591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.726561 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf"] Nov 02 14:28:19.727455 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:19.727429 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a665063_5a9e_4f57_add5_817979f6f382.slice/crio-356571f9cda4e0519cf80d31ad499e4b32a35089d22c1c2bb0a39f43f256043b WatchSource:0}: Error finding container 356571f9cda4e0519cf80d31ad499e4b32a35089d22c1c2bb0a39f43f256043b: Status 404 returned error can't find the container with id 356571f9cda4e0519cf80d31ad499e4b32a35089d22c1c2bb0a39f43f256043b Nov 02 14:28:19.867373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.867342 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" event={"ID":"4a665063-5a9e-4f57-add5-817979f6f382","Type":"ContainerStarted","Data":"356571f9cda4e0519cf80d31ad499e4b32a35089d22c1c2bb0a39f43f256043b"} Nov 02 14:28:19.868894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.868871 4431 generic.go:356] "Generic (PLEG): container finished" podID="a947cb83-0905-462f-8b99-dc2da8238f81" containerID="1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b" exitCode=0 Nov 02 14:28:19.869011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.868940 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" Nov 02 14:28:19.869011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.868956 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" event={"ID":"a947cb83-0905-462f-8b99-dc2da8238f81","Type":"ContainerDied","Data":"1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b"} Nov 02 14:28:19.869011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.868999 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr" event={"ID":"a947cb83-0905-462f-8b99-dc2da8238f81","Type":"ContainerDied","Data":"f4d743020aff813a06b270c6b9d013f74f936da9c37039ab5b2083ded95c49e8"} Nov 02 14:28:19.869124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.869030 4431 scope.go:122] "RemoveContainer" containerID="1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b" Nov 02 14:28:19.893131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.893113 4431 scope.go:122] "RemoveContainer" containerID="1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b" Nov 02 14:28:19.893422 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:19.893401 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b\": container with ID starting with 1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b not found: ID does not exist" containerID="1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b" Nov 02 14:28:19.893487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.893429 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b"} err="failed to get container status \"1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b\": rpc error: code = NotFound desc = could not find container \"1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b\": container with ID starting with 1b9a6d9d53c751931ce289ed5c16a55d8c08734e4aec81bf872952612cd3d35b not found: ID does not exist" Nov 02 14:28:19.936827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.936799 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr"] Nov 02 14:28:19.949805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:19.949772 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-6c865z6wr"] Nov 02 14:28:20.877416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:20.877333 4431 generic.go:356] "Generic (PLEG): container finished" podID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerID="25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c" exitCode=0 Nov 02 14:28:20.877918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:20.877406 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" event={"ID":"ff47094d-1dd4-4433-aaba-5dd88865d905","Type":"ContainerDied","Data":"25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c"} Nov 02 14:28:20.877918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:20.877907 4431 scope.go:122] "RemoveContainer" containerID="25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c" Nov 02 14:28:20.879023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:20.878985 4431 generic.go:356] "Generic (PLEG): container finished" podID="4a665063-5a9e-4f57-add5-817979f6f382" containerID="d73da1a04c423e53c84e3659961d8d16a5db2231fce4450f35b856060ceda4dc" exitCode=0 Nov 02 14:28:20.879143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:20.879064 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" event={"ID":"4a665063-5a9e-4f57-add5-817979f6f382","Type":"ContainerDied","Data":"d73da1a04c423e53c84e3659961d8d16a5db2231fce4450f35b856060ceda4dc"} Nov 02 14:28:20.879644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:20.879618 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:28:20.879736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:20.879697 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:28:20.879797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:20.879768 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:28:21.076317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:21.076285 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a947cb83-0905-462f-8b99-dc2da8238f81" path="/var/lib/kubelet/pods/a947cb83-0905-462f-8b99-dc2da8238f81/volumes" Nov 02 14:28:21.287866 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:21.287826 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:28:21.941833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:21.938979 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" event={"ID":"ff47094d-1dd4-4433-aaba-5dd88865d905","Type":"ContainerStarted","Data":"92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446"} Nov 02 14:28:22.160337 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.154845 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:22.251888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.251812 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-openshift-ca\") pod \"4a665063-5a9e-4f57-add5-817979f6f382\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " Nov 02 14:28:22.251888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.251864 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-xks-ca\") pod \"4a665063-5a9e-4f57-add5-817979f6f382\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " Nov 02 14:28:22.252124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.251905 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a665063-5a9e-4f57-add5-817979f6f382-kube-api-access-bpp7z\" (UniqueName: \"kubernetes.io/projected/4a665063-5a9e-4f57-add5-817979f6f382-kube-api-access-bpp7z\") pod \"4a665063-5a9e-4f57-add5-817979f6f382\" (UID: \"4a665063-5a9e-4f57-add5-817979f6f382\") " Nov 02 14:28:22.252271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.252244 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-openshift-ca" pod "4a665063-5a9e-4f57-add5-817979f6f382" (UID: "4a665063-5a9e-4f57-add5-817979f6f382"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:22.252419 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.252399 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:22.254096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.254066 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-xks-ca" pod "4a665063-5a9e-4f57-add5-817979f6f382" (UID: "4a665063-5a9e-4f57-add5-817979f6f382"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:22.259814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.254876 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a665063-5a9e-4f57-add5-817979f6f382-kube-api-access-bpp7z" pod "4a665063-5a9e-4f57-add5-817979f6f382" (UID: "4a665063-5a9e-4f57-add5-817979f6f382"). InnerVolumeSpecName "kube-api-access-bpp7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:22.353256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.353222 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpp7z\" (UniqueName: \"kubernetes.io/projected/4a665063-5a9e-4f57-add5-817979f6f382-kube-api-access-bpp7z\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:22.353256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.353254 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a665063-5a9e-4f57-add5-817979f6f382-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:22.844563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.844529 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:28:22.845688 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.845663 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 14:28:22.845765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.845713 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 14:28:22.946772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.946744 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" Nov 02 14:28:22.947236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.946738 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf" event={"ID":"4a665063-5a9e-4f57-add5-817979f6f382","Type":"ContainerDied","Data":"356571f9cda4e0519cf80d31ad499e4b32a35089d22c1c2bb0a39f43f256043b"} Nov 02 14:28:22.947236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:22.946855 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="356571f9cda4e0519cf80d31ad499e4b32a35089d22c1c2bb0a39f43f256043b" Nov 02 14:28:23.555760 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.555730 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv"] Nov 02 14:28:23.556308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.556291 4431 cpu_manager.go:414] "containerMap: removing container" podUID="4a665063-5a9e-4f57-add5-817979f6f382" containerName="revoke-keys" Nov 02 14:28:23.556368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.556309 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a665063-5a9e-4f57-add5-817979f6f382" containerName="revoke-keys" Nov 02 14:28:23.556438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.556427 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a665063-5a9e-4f57-add5-817979f6f382" containerName="revoke-keys" Nov 02 14:28:23.558721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.558706 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.595799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.592087 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"routing-overlay\"" Nov 02 14:28:23.595799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.592713 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"agc-cred-047606d198bb\"" Nov 02 14:28:23.595799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.592855 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-e2e--3ffe20a0-dockercfg-kwvdn\"" Nov 02 14:28:23.595799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.592951 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"kube-root-ca.crt\"" Nov 02 14:28:23.595799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.593012 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"openshift-service-ca.crt\"" Nov 02 14:28:23.595799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.593088 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-12b8da\"/\"payload-processing-external-model-plugins-e2e-extmod-b-12b8da\"" Nov 02 14:28:23.598139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.598114 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv"] Nov 02 14:28:23.691800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.670414 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6gm5\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-kube-api-access-z6gm5\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.691800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.670494 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.691800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.670594 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.691800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.670624 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.772615 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.772573 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gm5\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-kube-api-access-z6gm5\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.786321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.786288 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.818183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.818098 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.831798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.825877 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.831798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.819253 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.831798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.826954 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.831798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.815401 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.854799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.851936 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 14:28:23.854799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.852003 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 14:28:23.855748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.855322 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gm5\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-kube-api-access-z6gm5\") pod \"payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.893802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.889980 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:23.920976 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.920954 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl"] Nov 02 14:28:23.927950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.927888 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:23.930052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.929985 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl"] Nov 02 14:28:23.944632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.944487 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:28:23.944632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:23.944602 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:28:24.028953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.028891 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.029352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.028992 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpq2v\" (UniqueName: \"kubernetes.io/projected/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-kube-api-access-gpq2v\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.029352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.029064 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.129996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.129890 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpq2v\" (UniqueName: \"kubernetes.io/projected/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-kube-api-access-gpq2v\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.130191 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.130037 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.130267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.130253 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.135010 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.131830 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.170798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.153883 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.202816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.191871 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpq2v\" (UniqueName: \"kubernetes.io/projected/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-kube-api-access-gpq2v\") pod \"maas-api-revoke-sub-bf169b5f-c2bsl\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.256698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.256665 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:24.307048 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:24.306894 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5623896f_6ea6_489a_9057_466a89520539.slice/crio-38895a3fa697626a508be5c69f2224280a3c1280aec35dee42791a8c030ee58d WatchSource:0}: Error finding container 38895a3fa697626a508be5c69f2224280a3c1280aec35dee42791a8c030ee58d: Status 404 returned error can't find the container with id 38895a3fa697626a508be5c69f2224280a3c1280aec35dee42791a8c030ee58d Nov 02 14:28:24.307319 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.307299 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv"] Nov 02 14:28:24.414504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.414476 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47"] Nov 02 14:28:24.418291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.417999 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.449083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.448807 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47"] Nov 02 14:28:24.539798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.539551 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmzcj\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-kube-api-access-hmzcj\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.539798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.539706 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.539798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.539800 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.540137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.539898 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.631013 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:24.630979 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66580ea1_2b59_4661_b1e1_4bdcb206dd9a.slice/crio-da7ff4cadb468fc2937c6c5289cd36fa928ab7761d7b97f271ab1ff3fb36f858 WatchSource:0}: Error finding container da7ff4cadb468fc2937c6c5289cd36fa928ab7761d7b97f271ab1ff3fb36f858: Status 404 returned error can't find the container with id da7ff4cadb468fc2937c6c5289cd36fa928ab7761d7b97f271ab1ff3fb36f858 Nov 02 14:28:24.632034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.632002 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl"] Nov 02 14:28:24.641635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.641613 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.641717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.641688 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmzcj\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-kube-api-access-hmzcj\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.641816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.641768 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.641876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.641850 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.642542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.642520 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.642640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.642555 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.645046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.645023 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.666332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.666280 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmzcj\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-kube-api-access-hmzcj\") pod \"payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.733968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.733939 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:24.846337 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.845901 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 14:28:24.846337 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.845961 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 14:28:24.964425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.964335 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" event={"ID":"66580ea1-2b59-4661-b1e1-4bdcb206dd9a","Type":"ContainerStarted","Data":"da7ff4cadb468fc2937c6c5289cd36fa928ab7761d7b97f271ab1ff3fb36f858"} Nov 02 14:28:24.966078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.966046 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" event={"ID":"5623896f-6ea6-489a-9057-466a89520539","Type":"ContainerStarted","Data":"fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f"} Nov 02 14:28:24.966206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.966094 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" event={"ID":"5623896f-6ea6-489a-9057-466a89520539","Type":"ContainerStarted","Data":"38895a3fa697626a508be5c69f2224280a3c1280aec35dee42791a8c030ee58d"} Nov 02 14:28:24.966267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:24.966255 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:25.025040 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.025000 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" podStartSLOduration=2.024985587 podStartE2EDuration="2.024985587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:25.022046408 +0000 UTC m=+1870.059151340" watchObservedRunningTime="2026-11-02 14:28:25.024985587 +0000 UTC m=+1870.062090519" Nov 02 14:28:25.103458 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:25.103425 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e918fa_af8b_47b4_bf70_576844a8b0cc.slice/crio-40a5dc5889633d50c0ab30430dcb608d0e1d5f3027365ae4f9a35bde59a92c4f WatchSource:0}: Error finding container 40a5dc5889633d50c0ab30430dcb608d0e1d5f3027365ae4f9a35bde59a92c4f: Status 404 returned error can't find the container with id 40a5dc5889633d50c0ab30430dcb608d0e1d5f3027365ae4f9a35bde59a92c4f Nov 02 14:28:25.106755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.104723 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47"] Nov 02 14:28:25.845321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.845292 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 14:28:25.845477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.845340 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 14:28:25.997219 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.997180 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" event={"ID":"c6e918fa-af8b-47b4-bf70-576844a8b0cc","Type":"ContainerStarted","Data":"8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e"} Nov 02 14:28:25.997443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.997246 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:25.997443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.997270 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" event={"ID":"c6e918fa-af8b-47b4-bf70-576844a8b0cc","Type":"ContainerStarted","Data":"40a5dc5889633d50c0ab30430dcb608d0e1d5f3027365ae4f9a35bde59a92c4f"} Nov 02 14:28:25.999278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.999246 4431 generic.go:356] "Generic (PLEG): container finished" podID="66580ea1-2b59-4661-b1e1-4bdcb206dd9a" containerID="d6170728c05a1829f108b0da3b42ff633914b24d77e84827783f3db2ca82dc67" exitCode=0 Nov 02 14:28:25.999418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:25.999327 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" event={"ID":"66580ea1-2b59-4661-b1e1-4bdcb206dd9a","Type":"ContainerDied","Data":"d6170728c05a1829f108b0da3b42ff633914b24d77e84827783f3db2ca82dc67"} Nov 02 14:28:26.043493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:26.043435 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" podStartSLOduration=2.043420782 podStartE2EDuration="2.043420782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:26.041186217 +0000 UTC m=+1871.078291149" watchObservedRunningTime="2026-11-02 14:28:26.043420782 +0000 UTC m=+1871.080525739" Nov 02 14:28:26.845360 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:26.845319 4431 patch_prober.go:29] interesting pod/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 02 14:28:26.845768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:26.845378 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 02 14:28:27.147288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.147254 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:27.282713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.282674 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-xks-ca\") pod \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " Nov 02 14:28:27.282944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.282915 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-kube-api-access-gpq2v\" (UniqueName: \"kubernetes.io/projected/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-kube-api-access-gpq2v\") pod \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " Nov 02 14:28:27.283088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.283042 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-openshift-ca\") pod \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\" (UID: \"66580ea1-2b59-4661-b1e1-4bdcb206dd9a\") " Nov 02 14:28:27.284269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.284240 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-openshift-ca" pod "66580ea1-2b59-4661-b1e1-4bdcb206dd9a" (UID: "66580ea1-2b59-4661-b1e1-4bdcb206dd9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:27.286020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.285969 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-kube-api-access-gpq2v" pod "66580ea1-2b59-4661-b1e1-4bdcb206dd9a" (UID: "66580ea1-2b59-4661-b1e1-4bdcb206dd9a"). InnerVolumeSpecName "kube-api-access-gpq2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:27.286338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.286312 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-xks-ca" pod "66580ea1-2b59-4661-b1e1-4bdcb206dd9a" (UID: "66580ea1-2b59-4661-b1e1-4bdcb206dd9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:27.313868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.313846 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:28:27.384578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.384521 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:27.384578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.384543 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:27.384578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.384554 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpq2v\" (UniqueName: \"kubernetes.io/projected/66580ea1-2b59-4661-b1e1-4bdcb206dd9a-kube-api-access-gpq2v\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:27.849493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:27.849464 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:28:28.013755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:28.013722 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" Nov 02 14:28:28.013955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:28.013765 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl" event={"ID":"66580ea1-2b59-4661-b1e1-4bdcb206dd9a","Type":"ContainerDied","Data":"da7ff4cadb468fc2937c6c5289cd36fa928ab7761d7b97f271ab1ff3fb36f858"} Nov 02 14:28:28.013955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:28.013824 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da7ff4cadb468fc2937c6c5289cd36fa928ab7761d7b97f271ab1ff3fb36f858" Nov 02 14:28:28.018868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:28.018836 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:28:28.841463 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:28.841351 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8"] Nov 02 14:28:28.841694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:28.841662 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" containerID="cri-o://8c7de268a28655066b1081b212027956cfebeec13ef1734be85b9c7a172fd844" gracePeriod=30 Nov 02 14:28:31.118205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.118169 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4"] Nov 02 14:28:31.122821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.119151 4431 cpu_manager.go:414] "containerMap: removing container" podUID="66580ea1-2b59-4661-b1e1-4bdcb206dd9a" containerName="revoke-keys" Nov 02 14:28:31.122821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.119176 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66580ea1-2b59-4661-b1e1-4bdcb206dd9a" containerName="revoke-keys" Nov 02 14:28:31.122821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.119339 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="66580ea1-2b59-4661-b1e1-4bdcb206dd9a" containerName="revoke-keys" Nov 02 14:28:31.123436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.123412 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.144935 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.144911 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4"] Nov 02 14:28:31.146111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.146086 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f72bc6\"/\"kube-root-ca.crt\"" Nov 02 14:28:31.146222 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.146173 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f72bc6\"/\"openshift-service-ca.crt\"" Nov 02 14:28:31.154405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.154380 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f72bc6\"/\"default-dockercfg-n6pk7\"" Nov 02 14:28:31.154518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.154491 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f72bc6\"/\"simulator-w1-f72bc6-kserve-self-signed-certs\"" Nov 02 14:28:31.231792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.231739 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-dshm\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.231991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.231876 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-home\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.231991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.231938 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-model-cache\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.232117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.231977 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-tmp-dir\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.232283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.232260 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b062be-55b8-4b08-8512-e8c86b10e317-tls-certs\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.232378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.232289 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d67rr\" (UniqueName: \"kubernetes.io/projected/17b062be-55b8-4b08-8512-e8c86b10e317-kube-api-access-d67rr\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.261232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.261204 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt"] Nov 02 14:28:31.264762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.264738 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.299696 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.299662 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt"] Nov 02 14:28:31.303075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.303052 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:28:31.303249 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.303132 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:28:31.333899 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.333871 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-dshm\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.334168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.334148 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-home\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.334269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.334216 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-model-cache\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.334269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.334252 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-tmp-dir\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.334387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.334353 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b062be-55b8-4b08-8512-e8c86b10e317-tls-certs\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.334387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.334379 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67rr\" (UniqueName: \"kubernetes.io/projected/17b062be-55b8-4b08-8512-e8c86b10e317-kube-api-access-d67rr\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.335084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.335057 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-model-cache\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.335245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.335090 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-tmp-dir\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.335396 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.335376 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-home\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.337023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.336996 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-dshm\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.337786 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.337748 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b062be-55b8-4b08-8512-e8c86b10e317-tls-certs\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.361004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.360977 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67rr\" (UniqueName: \"kubernetes.io/projected/17b062be-55b8-4b08-8512-e8c86b10e317-kube-api-access-d67rr\") pod \"simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.438345 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.438304 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:31.439630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.439594 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.440000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.439981 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zqd\" (UniqueName: \"kubernetes.io/projected/9fa8f24c-0073-405a-be97-9575f08da07f-kube-api-access-c2zqd\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.440252 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.440229 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.544585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.542629 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.544585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.542763 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.544585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.542919 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zqd\" (UniqueName: \"kubernetes.io/projected/9fa8f24c-0073-405a-be97-9575f08da07f-kube-api-access-c2zqd\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.544585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.543359 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.550654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.550618 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.571097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.571070 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zqd\" (UniqueName: \"kubernetes.io/projected/9fa8f24c-0073-405a-be97-9575f08da07f-kube-api-access-c2zqd\") pod \"maas-api-revoke-sub-48fd8748-sqfkt\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.582772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.582575 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:31.777494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.777466 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4"] Nov 02 14:28:31.778831 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:31.778805 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17b062be_55b8_4b08_8512_e8c86b10e317.slice/crio-5e3ef1b848c1ee472fe7ae63171ac09767de3151bb2bf23755f839b3f4cdee53 WatchSource:0}: Error finding container 5e3ef1b848c1ee472fe7ae63171ac09767de3151bb2bf23755f839b3f4cdee53: Status 404 returned error can't find the container with id 5e3ef1b848c1ee472fe7ae63171ac09767de3151bb2bf23755f839b3f4cdee53 Nov 02 14:28:31.908107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:31.908082 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt"] Nov 02 14:28:31.910113 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:31.910085 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa8f24c_0073_405a_be97_9575f08da07f.slice/crio-7f7638d8cbbae8ddcaf81042c52f1fab0973a0473341da2af96eb619ea31a64e WatchSource:0}: Error finding container 7f7638d8cbbae8ddcaf81042c52f1fab0973a0473341da2af96eb619ea31a64e: Status 404 returned error can't find the container with id 7f7638d8cbbae8ddcaf81042c52f1fab0973a0473341da2af96eb619ea31a64e Nov 02 14:28:32.047177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:32.047100 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" event={"ID":"17b062be-55b8-4b08-8512-e8c86b10e317","Type":"ContainerStarted","Data":"d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2"} Nov 02 14:28:32.047177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:32.047146 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" event={"ID":"17b062be-55b8-4b08-8512-e8c86b10e317","Type":"ContainerStarted","Data":"5e3ef1b848c1ee472fe7ae63171ac09767de3151bb2bf23755f839b3f4cdee53"} Nov 02 14:28:32.048468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:32.048437 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" event={"ID":"9fa8f24c-0073-405a-be97-9575f08da07f","Type":"ContainerStarted","Data":"7f7638d8cbbae8ddcaf81042c52f1fab0973a0473341da2af96eb619ea31a64e"} Nov 02 14:28:32.097132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:32.097088 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" podStartSLOduration=1.097075359 podStartE2EDuration="1.097075359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:32.095239686 +0000 UTC m=+1877.132344619" watchObservedRunningTime="2026-11-02 14:28:32.097075359 +0000 UTC m=+1877.134180288" Nov 02 14:28:32.299145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:32.298936 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:28:32.439399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:32.439372 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:33.057210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:33.057116 4431 generic.go:356] "Generic (PLEG): container finished" podID="9fa8f24c-0073-405a-be97-9575f08da07f" containerID="2eea8f0b71b577fed291e2dda8907d0682512decc2a976ecaacbb28346a7350e" exitCode=0 Nov 02 14:28:33.057367 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:33.057198 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" event={"ID":"9fa8f24c-0073-405a-be97-9575f08da07f","Type":"ContainerDied","Data":"2eea8f0b71b577fed291e2dda8907d0682512decc2a976ecaacbb28346a7350e"} Nov 02 14:28:34.067660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.067628 4431 generic.go:356] "Generic (PLEG): container finished" podID="a947c82c-52dc-487d-a500-d6402419accb" containerID="8c7de268a28655066b1081b212027956cfebeec13ef1734be85b9c7a172fd844" exitCode=0 Nov 02 14:28:34.068211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.067705 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" event={"ID":"a947c82c-52dc-487d-a500-d6402419accb","Type":"ContainerDied","Data":"8c7de268a28655066b1081b212027956cfebeec13ef1734be85b9c7a172fd844"} Nov 02 14:28:34.139059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.139033 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:28:34.243997 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.243972 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:34.279808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.279749 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-socket\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.279947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.279884 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-certs\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.280018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.279985 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-socket" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:34.280018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.279995 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-credential-socket\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.280142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280034 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-kube-api-access-sxv6f\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-kube-api-access-sxv6f\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.280142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280084 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-istio-token\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-istio-token\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.280244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280147 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-envoy\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.280244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280184 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-certs" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:34.280244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280217 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-credential-socket" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:34.280414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280266 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a947c82c-52dc-487d-a500-d6402419accb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a947c82c-52dc-487d-a500-d6402419accb-istio-podinfo\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.280414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280300 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a947c82c-52dc-487d-a500-d6402419accb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a947c82c-52dc-487d-a500-d6402419accb-istiod-ca-cert\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.280414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280338 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-data\") pod \"a947c82c-52dc-487d-a500-d6402419accb\" (UID: \"a947c82c-52dc-487d-a500-d6402419accb\") " Nov 02 14:28:34.280727 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280652 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-data" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:34.280727 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280658 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a947c82c-52dc-487d-a500-d6402419accb-istiod-ca-cert" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:34.280970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280885 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a947c82c-52dc-487d-a500-d6402419accb-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.280970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280906 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.280970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.280920 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.281132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.281007 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.281132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.281026 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.282675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.282653 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-istio-token" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:34.282816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.282756 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-kube-api-access-sxv6f" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "kube-api-access-sxv6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:34.282913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.282848 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-envoy" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:28:34.282965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.282922 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a947c82c-52dc-487d-a500-d6402419accb-istio-podinfo" pod "a947c82c-52dc-487d-a500-d6402419accb" (UID: "a947c82c-52dc-487d-a500-d6402419accb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:28:34.381592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.381513 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa8f24c-0073-405a-be97-9575f08da07f-kube-api-access-c2zqd\" (UniqueName: \"kubernetes.io/projected/9fa8f24c-0073-405a-be97-9575f08da07f-kube-api-access-c2zqd\") pod \"9fa8f24c-0073-405a-be97-9575f08da07f\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " Nov 02 14:28:34.381592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.381566 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-openshift-ca\") pod \"9fa8f24c-0073-405a-be97-9575f08da07f\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " Nov 02 14:28:34.381898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.381878 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-xks-ca\") pod \"9fa8f24c-0073-405a-be97-9575f08da07f\" (UID: \"9fa8f24c-0073-405a-be97-9575f08da07f\") " Nov 02 14:28:34.382024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.381986 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-openshift-ca" pod "9fa8f24c-0073-405a-be97-9575f08da07f" (UID: "9fa8f24c-0073-405a-be97-9575f08da07f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:34.382380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.382354 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxv6f\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-kube-api-access-sxv6f\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.382380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.382376 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a947c82c-52dc-487d-a500-d6402419accb-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.382595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.382390 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a947c82c-52dc-487d-a500-d6402419accb-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.382595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.382404 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a947c82c-52dc-487d-a500-d6402419accb-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.382595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.382417 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.383852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.383828 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa8f24c-0073-405a-be97-9575f08da07f-kube-api-access-c2zqd" pod "9fa8f24c-0073-405a-be97-9575f08da07f" (UID: "9fa8f24c-0073-405a-be97-9575f08da07f"). InnerVolumeSpecName "kube-api-access-c2zqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:34.384016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.384000 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-xks-ca" pod "9fa8f24c-0073-405a-be97-9575f08da07f" (UID: "9fa8f24c-0073-405a-be97-9575f08da07f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:34.483690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.483656 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa8f24c-0073-405a-be97-9575f08da07f-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.483690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.483682 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2zqd\" (UniqueName: \"kubernetes.io/projected/9fa8f24c-0073-405a-be97-9575f08da07f-kube-api-access-c2zqd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:34.725056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.725025 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq"] Nov 02 14:28:34.725748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.725731 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" Nov 02 14:28:34.725748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.725749 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" Nov 02 14:28:34.725894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.725763 4431 cpu_manager.go:414] "containerMap: removing container" podUID="9fa8f24c-0073-405a-be97-9575f08da07f" containerName="revoke-keys" Nov 02 14:28:34.725894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.725768 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa8f24c-0073-405a-be97-9575f08da07f" containerName="revoke-keys" Nov 02 14:28:34.725975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.725959 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa8f24c-0073-405a-be97-9575f08da07f" containerName="revoke-keys" Nov 02 14:28:34.726009 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.725995 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a947c82c-52dc-487d-a500-d6402419accb" containerName="istio-proxy" Nov 02 14:28:34.729596 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.729576 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:34.748575 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.748547 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq"] Nov 02 14:28:34.888554 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.888517 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzdvb\" (UniqueName: \"kubernetes.io/projected/14a285e2-6dc7-41e2-8057-0a411ab404df-kube-api-access-tzdvb\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:34.888698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.888663 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:34.888752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.888738 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:34.989826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.989755 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:34.989950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.989899 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdvb\" (UniqueName: \"kubernetes.io/projected/14a285e2-6dc7-41e2-8057-0a411ab404df-kube-api-access-tzdvb\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:34.989993 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.989962 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:34.990406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.990384 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:34.992302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:34.992283 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:35.022762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.022737 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzdvb\" (UniqueName: \"kubernetes.io/projected/14a285e2-6dc7-41e2-8057-0a411ab404df-kube-api-access-tzdvb\") pod \"maas-api-revoke-sub-8b08f0db-qkcwq\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:35.043556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.043537 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:35.079548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.079497 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" Nov 02 14:28:35.080020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.079507 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8" event={"ID":"a947c82c-52dc-487d-a500-d6402419accb","Type":"ContainerDied","Data":"ec42bfa5279943cf1060f6ec7b0d577cfa158c7651fe64b595f55b2bcea98e0d"} Nov 02 14:28:35.080020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.079666 4431 scope.go:122] "RemoveContainer" containerID="8c7de268a28655066b1081b212027956cfebeec13ef1734be85b9c7a172fd844" Nov 02 14:28:35.081526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.081505 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" event={"ID":"9fa8f24c-0073-405a-be97-9575f08da07f","Type":"ContainerDied","Data":"7f7638d8cbbae8ddcaf81042c52f1fab0973a0473341da2af96eb619ea31a64e"} Nov 02 14:28:35.081655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.081544 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f7638d8cbbae8ddcaf81042c52f1fab0973a0473341da2af96eb619ea31a64e" Nov 02 14:28:35.081655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.081603 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt" Nov 02 14:28:35.169918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.169888 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8"] Nov 02 14:28:35.182440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.182412 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-623b4b-openshift-default-5d6db7b47-xkxn8"] Nov 02 14:28:35.356900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:35.356874 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq"] Nov 02 14:28:35.358768 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:35.358743 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a285e2_6dc7_41e2_8057_0a411ab404df.slice/crio-c138cf53b1e5bf18315a3a0a32f50126e14156b81748950ab29e1a8cb3089919 WatchSource:0}: Error finding container c138cf53b1e5bf18315a3a0a32f50126e14156b81748950ab29e1a8cb3089919: Status 404 returned error can't find the container with id c138cf53b1e5bf18315a3a0a32f50126e14156b81748950ab29e1a8cb3089919 Nov 02 14:28:36.002500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:36.002419 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:36.091742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:36.091709 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" event={"ID":"14a285e2-6dc7-41e2-8057-0a411ab404df","Type":"ContainerStarted","Data":"39060a90914e61871438c701a3114b1a09703d9a6701ec9cc5bac969ba117010"} Nov 02 14:28:36.092172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:36.091758 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" event={"ID":"14a285e2-6dc7-41e2-8057-0a411ab404df","Type":"ContainerStarted","Data":"c138cf53b1e5bf18315a3a0a32f50126e14156b81748950ab29e1a8cb3089919"} Nov 02 14:28:36.136045 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:36.135994 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" podStartSLOduration=2.135981711 podStartE2EDuration="2.135981711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:36.133977974 +0000 UTC m=+1881.171082908" watchObservedRunningTime="2026-11-02 14:28:36.135981711 +0000 UTC m=+1881.173086643" Nov 02 14:28:37.008822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:37.008768 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:37.083835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:37.083801 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a947c82c-52dc-487d-a500-d6402419accb" path="/var/lib/kubelet/pods/a947c82c-52dc-487d-a500-d6402419accb/volumes" Nov 02 14:28:37.096981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:37.096053 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc"] Nov 02 14:28:37.103712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:37.103672 4431 generic.go:356] "Generic (PLEG): container finished" podID="14a285e2-6dc7-41e2-8057-0a411ab404df" containerID="39060a90914e61871438c701a3114b1a09703d9a6701ec9cc5bac969ba117010" exitCode=0 Nov 02 14:28:37.103902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:37.103855 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" event={"ID":"14a285e2-6dc7-41e2-8057-0a411ab404df","Type":"ContainerDied","Data":"39060a90914e61871438c701a3114b1a09703d9a6701ec9cc5bac969ba117010"} Nov 02 14:28:38.253518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.253494 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:38.428765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.428708 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a285e2-6dc7-41e2-8057-0a411ab404df-kube-api-access-tzdvb\" (UniqueName: \"kubernetes.io/projected/14a285e2-6dc7-41e2-8057-0a411ab404df-kube-api-access-tzdvb\") pod \"14a285e2-6dc7-41e2-8057-0a411ab404df\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " Nov 02 14:28:38.428949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.428922 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-openshift-ca\") pod \"14a285e2-6dc7-41e2-8057-0a411ab404df\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " Nov 02 14:28:38.428994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.428953 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-xks-ca\") pod \"14a285e2-6dc7-41e2-8057-0a411ab404df\" (UID: \"14a285e2-6dc7-41e2-8057-0a411ab404df\") " Nov 02 14:28:38.429316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.429288 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-openshift-ca" pod "14a285e2-6dc7-41e2-8057-0a411ab404df" (UID: "14a285e2-6dc7-41e2-8057-0a411ab404df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:38.429457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.429439 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:38.431112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.431085 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-xks-ca" pod "14a285e2-6dc7-41e2-8057-0a411ab404df" (UID: "14a285e2-6dc7-41e2-8057-0a411ab404df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:38.431224 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.431110 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a285e2-6dc7-41e2-8057-0a411ab404df-kube-api-access-tzdvb" pod "14a285e2-6dc7-41e2-8057-0a411ab404df" (UID: "14a285e2-6dc7-41e2-8057-0a411ab404df"). InnerVolumeSpecName "kube-api-access-tzdvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:38.530844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.530818 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a285e2-6dc7-41e2-8057-0a411ab404df-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:38.530844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:38.530841 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzdvb\" (UniqueName: \"kubernetes.io/projected/14a285e2-6dc7-41e2-8057-0a411ab404df-kube-api-access-tzdvb\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:39.118042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:39.118012 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" event={"ID":"14a285e2-6dc7-41e2-8057-0a411ab404df","Type":"ContainerDied","Data":"c138cf53b1e5bf18315a3a0a32f50126e14156b81748950ab29e1a8cb3089919"} Nov 02 14:28:39.118042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:39.118034 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq" Nov 02 14:28:39.118235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:39.118056 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c138cf53b1e5bf18315a3a0a32f50126e14156b81748950ab29e1a8cb3089919" Nov 02 14:28:41.439354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:41.439321 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:41.454587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:41.454551 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:42.078245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.078213 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m"] Nov 02 14:28:42.078732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.078720 4431 cpu_manager.go:414] "containerMap: removing container" podUID="14a285e2-6dc7-41e2-8057-0a411ab404df" containerName="revoke-keys" Nov 02 14:28:42.078793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.078734 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a285e2-6dc7-41e2-8057-0a411ab404df" containerName="revoke-keys" Nov 02 14:28:42.078885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.078876 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a285e2-6dc7-41e2-8057-0a411ab404df" containerName="revoke-keys" Nov 02 14:28:42.081162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.081144 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.097003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.096970 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" podUID="e6aaa6f9-172c-48fe-96ac-b29f4a554364" containerName="payload-processing" containerID="cri-o://06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439" gracePeriod=25 Nov 02 14:28:42.098167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.098137 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:28:42.102758 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.102737 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m"] Nov 02 14:28:42.111154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.111135 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:28:42.148158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.148137 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:28:42.270965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.270929 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.271147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.271125 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkx9c\" (UniqueName: \"kubernetes.io/projected/6ad23ad4-f234-4b62-8d57-42547fac459a-kube-api-access-xkx9c\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.271351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.271333 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.364983 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.364955 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:42.372760 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.372732 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.372892 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.372873 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.373048 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.373029 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkx9c\" (UniqueName: \"kubernetes.io/projected/6ad23ad4-f234-4b62-8d57-42547fac459a-kube-api-access-xkx9c\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.373693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.373474 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.379176 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.379152 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.398080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.398058 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkx9c\" (UniqueName: \"kubernetes.io/projected/6ad23ad4-f234-4b62-8d57-42547fac459a-kube-api-access-xkx9c\") pod \"maas-api-revoke-sub-6242810d-5zj6m\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.474347 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.474320 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-routing-overlay-volume\") pod \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " Nov 02 14:28:42.474721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.474457 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-provider-credentials\") pod \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " Nov 02 14:28:42.474721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.474476 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-plugins-config-volume\") pod \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " Nov 02 14:28:42.474721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.474544 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-kube-api-access-bsghm\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-kube-api-access-bsghm\") pod \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\" (UID: \"e6aaa6f9-172c-48fe-96ac-b29f4a554364\") " Nov 02 14:28:42.474721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.474657 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-routing-overlay-volume" pod "e6aaa6f9-172c-48fe-96ac-b29f4a554364" (UID: "e6aaa6f9-172c-48fe-96ac-b29f4a554364"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:42.475003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.474979 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:42.475049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.474996 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-plugins-config-volume" pod "e6aaa6f9-172c-48fe-96ac-b29f4a554364" (UID: "e6aaa6f9-172c-48fe-96ac-b29f4a554364"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:42.476471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.476450 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-provider-credentials" pod "e6aaa6f9-172c-48fe-96ac-b29f4a554364" (UID: "e6aaa6f9-172c-48fe-96ac-b29f4a554364"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:42.476568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.476548 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-kube-api-access-bsghm" pod "e6aaa6f9-172c-48fe-96ac-b29f4a554364" (UID: "e6aaa6f9-172c-48fe-96ac-b29f4a554364"). InnerVolumeSpecName "kube-api-access-bsghm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:42.575813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.575749 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsghm\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-kube-api-access-bsghm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:42.575813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.575809 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6aaa6f9-172c-48fe-96ac-b29f4a554364-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:42.576057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.575826 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6aaa6f9-172c-48fe-96ac-b29f4a554364-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:42.693064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.693038 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:42.995357 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:42.995330 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m"] Nov 02 14:28:42.995993 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:42.995957 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad23ad4_f234_4b62_8d57_42547fac459a.slice/crio-682721741af28a0f5cc54081009683b6d2e52ae4146bf4bddb837731469c694b WatchSource:0}: Error finding container 682721741af28a0f5cc54081009683b6d2e52ae4146bf4bddb837731469c694b: Status 404 returned error can't find the container with id 682721741af28a0f5cc54081009683b6d2e52ae4146bf4bddb837731469c694b Nov 02 14:28:43.143382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.143354 4431 generic.go:356] "Generic (PLEG): container finished" podID="e6aaa6f9-172c-48fe-96ac-b29f4a554364" containerID="06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439" exitCode=0 Nov 02 14:28:43.143497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.143400 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" Nov 02 14:28:43.143497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.143438 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" event={"ID":"e6aaa6f9-172c-48fe-96ac-b29f4a554364","Type":"ContainerDied","Data":"06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439"} Nov 02 14:28:43.143497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.143492 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc" event={"ID":"e6aaa6f9-172c-48fe-96ac-b29f4a554364","Type":"ContainerDied","Data":"919b2e8d6b538c1739a1594827224104f4fe7f703c051a24efcece02d5994fc1"} Nov 02 14:28:43.143659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.143524 4431 scope.go:122] "RemoveContainer" containerID="06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439" Nov 02 14:28:43.145071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.145049 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" event={"ID":"6ad23ad4-f234-4b62-8d57-42547fac459a","Type":"ContainerStarted","Data":"682721741af28a0f5cc54081009683b6d2e52ae4146bf4bddb837731469c694b"} Nov 02 14:28:43.153652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.153622 4431 scope.go:122] "RemoveContainer" containerID="06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439" Nov 02 14:28:43.153940 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:43.153922 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439\": container with ID starting with 06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439 not found: ID does not exist" containerID="06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439" Nov 02 14:28:43.154003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.153954 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439"} err="failed to get container status \"06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439\": rpc error: code = NotFound desc = could not find container \"06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439\": container with ID starting with 06928f042b8edfca48e30c2df4321a9cf7bc8584c264cb9b3cd9ea03a3b92439 not found: ID does not exist" Nov 02 14:28:43.205735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.205708 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc"] Nov 02 14:28:43.218539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.218518 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-7bc94jhjc"] Nov 02 14:28:43.628722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:43.628651 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4"] Nov 02 14:28:44.153643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:44.153607 4431 generic.go:356] "Generic (PLEG): container finished" podID="6ad23ad4-f234-4b62-8d57-42547fac459a" containerID="a13488aa2f5c2dbc2068b5e7eae21895a5af981b2af0236d272df6f1412e639b" exitCode=0 Nov 02 14:28:44.153844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:44.153698 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" event={"ID":"6ad23ad4-f234-4b62-8d57-42547fac459a","Type":"ContainerDied","Data":"a13488aa2f5c2dbc2068b5e7eae21895a5af981b2af0236d272df6f1412e639b"} Nov 02 14:28:45.084733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.084703 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6aaa6f9-172c-48fe-96ac-b29f4a554364" path="/var/lib/kubelet/pods/e6aaa6f9-172c-48fe-96ac-b29f4a554364/volumes" Nov 02 14:28:45.322863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.322842 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:45.407618 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.407585 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-xks-ca\") pod \"6ad23ad4-f234-4b62-8d57-42547fac459a\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " Nov 02 14:28:45.407848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.407641 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad23ad4-f234-4b62-8d57-42547fac459a-kube-api-access-xkx9c\" (UniqueName: \"kubernetes.io/projected/6ad23ad4-f234-4b62-8d57-42547fac459a-kube-api-access-xkx9c\") pod \"6ad23ad4-f234-4b62-8d57-42547fac459a\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " Nov 02 14:28:45.407848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.407717 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-openshift-ca\") pod \"6ad23ad4-f234-4b62-8d57-42547fac459a\" (UID: \"6ad23ad4-f234-4b62-8d57-42547fac459a\") " Nov 02 14:28:45.408110 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.408077 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-openshift-ca" pod "6ad23ad4-f234-4b62-8d57-42547fac459a" (UID: "6ad23ad4-f234-4b62-8d57-42547fac459a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:45.409869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.409844 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad23ad4-f234-4b62-8d57-42547fac459a-kube-api-access-xkx9c" pod "6ad23ad4-f234-4b62-8d57-42547fac459a" (UID: "6ad23ad4-f234-4b62-8d57-42547fac459a"). InnerVolumeSpecName "kube-api-access-xkx9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:45.410227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.410206 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-xks-ca" pod "6ad23ad4-f234-4b62-8d57-42547fac459a" (UID: "6ad23ad4-f234-4b62-8d57-42547fac459a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:45.509021 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.508983 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:45.509021 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.509011 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkx9c\" (UniqueName: \"kubernetes.io/projected/6ad23ad4-f234-4b62-8d57-42547fac459a-kube-api-access-xkx9c\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:45.509021 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.509022 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad23ad4-f234-4b62-8d57-42547fac459a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:45.561559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.561524 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g"] Nov 02 14:28:45.562130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.562113 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e6aaa6f9-172c-48fe-96ac-b29f4a554364" containerName="payload-processing" Nov 02 14:28:45.562130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.562129 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6aaa6f9-172c-48fe-96ac-b29f4a554364" containerName="payload-processing" Nov 02 14:28:45.562298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.562150 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6ad23ad4-f234-4b62-8d57-42547fac459a" containerName="revoke-keys" Nov 02 14:28:45.562298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.562159 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad23ad4-f234-4b62-8d57-42547fac459a" containerName="revoke-keys" Nov 02 14:28:45.562298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.562253 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad23ad4-f234-4b62-8d57-42547fac459a" containerName="revoke-keys" Nov 02 14:28:45.562298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.562270 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6aaa6f9-172c-48fe-96ac-b29f4a554364" containerName="payload-processing" Nov 02 14:28:45.564759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.564739 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.589267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.589240 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g"] Nov 02 14:28:45.609811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.609765 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswtc\" (UniqueName: \"kubernetes.io/projected/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-kube-api-access-nswtc\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.609964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.609938 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.610065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.610045 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.711198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.711129 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswtc\" (UniqueName: \"kubernetes.io/projected/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-kube-api-access-nswtc\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.711198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.711186 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.711365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.711232 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.711989 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.711966 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.713652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.713623 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.735568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.735537 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswtc\" (UniqueName: \"kubernetes.io/projected/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-kube-api-access-nswtc\") pod \"maas-api-revoke-sub-33376e24-f726g\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:45.877882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:45.877848 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:46.166847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:46.166821 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" Nov 02 14:28:46.167353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:46.166816 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m" event={"ID":"6ad23ad4-f234-4b62-8d57-42547fac459a","Type":"ContainerDied","Data":"682721741af28a0f5cc54081009683b6d2e52ae4146bf4bddb837731469c694b"} Nov 02 14:28:46.167353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:46.166948 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682721741af28a0f5cc54081009683b6d2e52ae4146bf4bddb837731469c694b" Nov 02 14:28:46.179240 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:46.179214 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba85d83_ea10_41e4_8507_e9fbd2bba1a1.slice/crio-d29b14eb165c03fb42b860c8d34cd720d08fa9a4c0c8708055834b6845f4c62c WatchSource:0}: Error finding container d29b14eb165c03fb42b860c8d34cd720d08fa9a4c0c8708055834b6845f4c62c: Status 404 returned error can't find the container with id d29b14eb165c03fb42b860c8d34cd720d08fa9a4c0c8708055834b6845f4c62c Nov 02 14:28:46.179390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:46.179332 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g"] Nov 02 14:28:46.453732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:46.453636 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47"] Nov 02 14:28:47.183758 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:47.183722 4431 generic.go:356] "Generic (PLEG): container finished" podID="dba85d83-ea10-41e4-8507-e9fbd2bba1a1" containerID="9f1cc4897174ecdf47b8474ec6cde55cdb577e0b89965a0c57979f82052f4d5f" exitCode=0 Nov 02 14:28:47.184215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:47.183801 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" event={"ID":"dba85d83-ea10-41e4-8507-e9fbd2bba1a1","Type":"ContainerDied","Data":"9f1cc4897174ecdf47b8474ec6cde55cdb577e0b89965a0c57979f82052f4d5f"} Nov 02 14:28:47.184215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:47.183869 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" event={"ID":"dba85d83-ea10-41e4-8507-e9fbd2bba1a1","Type":"ContainerStarted","Data":"d29b14eb165c03fb42b860c8d34cd720d08fa9a4c0c8708055834b6845f4c62c"} Nov 02 14:28:47.495513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:47.495480 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv"] Nov 02 14:28:48.333907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.333884 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:48.355673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.355649 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-xks-ca\") pod \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " Nov 02 14:28:48.355845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.355711 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-kube-api-access-nswtc\" (UniqueName: \"kubernetes.io/projected/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-kube-api-access-nswtc\") pod \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " Nov 02 14:28:48.355916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.355853 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-openshift-ca\") pod \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\" (UID: \"dba85d83-ea10-41e4-8507-e9fbd2bba1a1\") " Nov 02 14:28:48.356251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.356227 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-openshift-ca" pod "dba85d83-ea10-41e4-8507-e9fbd2bba1a1" (UID: "dba85d83-ea10-41e4-8507-e9fbd2bba1a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:48.356378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.356362 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:48.357852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.357826 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-xks-ca" pod "dba85d83-ea10-41e4-8507-e9fbd2bba1a1" (UID: "dba85d83-ea10-41e4-8507-e9fbd2bba1a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:48.357961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.357931 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-kube-api-access-nswtc" pod "dba85d83-ea10-41e4-8507-e9fbd2bba1a1" (UID: "dba85d83-ea10-41e4-8507-e9fbd2bba1a1"). InnerVolumeSpecName "kube-api-access-nswtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:48.457881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.457824 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:48.457881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.457847 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nswtc\" (UniqueName: \"kubernetes.io/projected/dba85d83-ea10-41e4-8507-e9fbd2bba1a1-kube-api-access-nswtc\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:48.629659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.629628 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" podUID="0ff16766-b8c1-4bab-83c5-a4732026f528" containerName="payload-processing" containerID="cri-o://934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3" gracePeriod=25 Nov 02 14:28:48.888736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.888710 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:28:48.963745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.963715 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-routing-overlay-volume\") pod \"0ff16766-b8c1-4bab-83c5-a4732026f528\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " Nov 02 14:28:48.963912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.963765 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-kube-api-access-ghl55\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-kube-api-access-ghl55\") pod \"0ff16766-b8c1-4bab-83c5-a4732026f528\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " Nov 02 14:28:48.963972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.963937 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-plugins-config-volume\") pod \"0ff16766-b8c1-4bab-83c5-a4732026f528\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " Nov 02 14:28:48.964019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.963979 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-provider-credentials\") pod \"0ff16766-b8c1-4bab-83c5-a4732026f528\" (UID: \"0ff16766-b8c1-4bab-83c5-a4732026f528\") " Nov 02 14:28:48.964270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.964239 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-routing-overlay-volume" pod "0ff16766-b8c1-4bab-83c5-a4732026f528" (UID: "0ff16766-b8c1-4bab-83c5-a4732026f528"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:48.964420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.964363 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-plugins-config-volume" pod "0ff16766-b8c1-4bab-83c5-a4732026f528" (UID: "0ff16766-b8c1-4bab-83c5-a4732026f528"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:48.964483 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.964431 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:48.966065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.966036 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-kube-api-access-ghl55" pod "0ff16766-b8c1-4bab-83c5-a4732026f528" (UID: "0ff16766-b8c1-4bab-83c5-a4732026f528"). InnerVolumeSpecName "kube-api-access-ghl55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:48.966065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:48.966057 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-provider-credentials" pod "0ff16766-b8c1-4bab-83c5-a4732026f528" (UID: "0ff16766-b8c1-4bab-83c5-a4732026f528"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:49.065614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.065550 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff16766-b8c1-4bab-83c5-a4732026f528-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:49.065614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.065575 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:49.065614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.065585 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghl55\" (UniqueName: \"kubernetes.io/projected/0ff16766-b8c1-4bab-83c5-a4732026f528-kube-api-access-ghl55\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:49.199185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.199156 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" Nov 02 14:28:49.199185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.199162 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g" event={"ID":"dba85d83-ea10-41e4-8507-e9fbd2bba1a1","Type":"ContainerDied","Data":"d29b14eb165c03fb42b860c8d34cd720d08fa9a4c0c8708055834b6845f4c62c"} Nov 02 14:28:49.199432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.199210 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d29b14eb165c03fb42b860c8d34cd720d08fa9a4c0c8708055834b6845f4c62c" Nov 02 14:28:49.200684 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.200656 4431 generic.go:356] "Generic (PLEG): container finished" podID="0ff16766-b8c1-4bab-83c5-a4732026f528" containerID="934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3" exitCode=0 Nov 02 14:28:49.200860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.200696 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" Nov 02 14:28:49.200860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.200731 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" event={"ID":"0ff16766-b8c1-4bab-83c5-a4732026f528","Type":"ContainerDied","Data":"934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3"} Nov 02 14:28:49.200860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.200789 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4" event={"ID":"0ff16766-b8c1-4bab-83c5-a4732026f528","Type":"ContainerDied","Data":"7192a143f7cd1f837800f6009a7cd2cb82773d9af0be2fbb7edbb3473cb4355f"} Nov 02 14:28:49.200860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.200817 4431 scope.go:122] "RemoveContainer" containerID="934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3" Nov 02 14:28:49.210320 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.210304 4431 scope.go:122] "RemoveContainer" containerID="934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3" Nov 02 14:28:49.210565 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:49.210549 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3\": container with ID starting with 934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3 not found: ID does not exist" containerID="934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3" Nov 02 14:28:49.210626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.210573 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3"} err="failed to get container status \"934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3\": rpc error: code = NotFound desc = could not find container \"934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3\": container with ID starting with 934e8d8092ed94770eaefdbe51f739ed98115b0db928b03f8cf598fd17799ff3 not found: ID does not exist" Nov 02 14:28:49.257537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.257520 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4"] Nov 02 14:28:49.262085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:49.262069 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-649chdmp4"] Nov 02 14:28:51.077752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.077717 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff16766-b8c1-4bab-83c5-a4732026f528" path="/var/lib/kubelet/pods/0ff16766-b8c1-4bab-83c5-a4732026f528/volumes" Nov 02 14:28:51.285685 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:51.285648 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:28:51.454909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.454876 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" podUID="c6e918fa-af8b-47b4-bf70-576844a8b0cc" containerName="payload-processing" containerID="cri-o://8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e" gracePeriod=25 Nov 02 14:28:51.728569 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.728546 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:51.793667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.793638 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-provider-credentials\") pod \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " Nov 02 14:28:51.793813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.793728 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-plugins-config-volume\") pod \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " Nov 02 14:28:51.793917 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.793894 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-kube-api-access-hmzcj\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-kube-api-access-hmzcj\") pod \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " Nov 02 14:28:51.793999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.793963 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-routing-overlay-volume\") pod \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\" (UID: \"c6e918fa-af8b-47b4-bf70-576844a8b0cc\") " Nov 02 14:28:51.794223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.794180 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-plugins-config-volume" pod "c6e918fa-af8b-47b4-bf70-576844a8b0cc" (UID: "c6e918fa-af8b-47b4-bf70-576844a8b0cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:51.794497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.794469 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-routing-overlay-volume" pod "c6e918fa-af8b-47b4-bf70-576844a8b0cc" (UID: "c6e918fa-af8b-47b4-bf70-576844a8b0cc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:51.794680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.794660 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:51.794755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.794685 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6e918fa-af8b-47b4-bf70-576844a8b0cc-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:51.795948 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.795926 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-provider-credentials" pod "c6e918fa-af8b-47b4-bf70-576844a8b0cc" (UID: "c6e918fa-af8b-47b4-bf70-576844a8b0cc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:51.796122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.796099 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-kube-api-access-hmzcj" pod "c6e918fa-af8b-47b4-bf70-576844a8b0cc" (UID: "c6e918fa-af8b-47b4-bf70-576844a8b0cc"). InnerVolumeSpecName "kube-api-access-hmzcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:51.895400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.895376 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmzcj\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-kube-api-access-hmzcj\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:51.895400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:51.895399 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6e918fa-af8b-47b4-bf70-576844a8b0cc-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:52.223901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.223869 4431 generic.go:356] "Generic (PLEG): container finished" podID="c6e918fa-af8b-47b4-bf70-576844a8b0cc" containerID="8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e" exitCode=0 Nov 02 14:28:52.224309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.223939 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" event={"ID":"c6e918fa-af8b-47b4-bf70-576844a8b0cc","Type":"ContainerDied","Data":"8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e"} Nov 02 14:28:52.224309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.223986 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" event={"ID":"c6e918fa-af8b-47b4-bf70-576844a8b0cc","Type":"ContainerDied","Data":"40a5dc5889633d50c0ab30430dcb608d0e1d5f3027365ae4f9a35bde59a92c4f"} Nov 02 14:28:52.224309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.224014 4431 scope.go:122] "RemoveContainer" containerID="8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e" Nov 02 14:28:52.224309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.224131 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47" Nov 02 14:28:52.234756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.234736 4431 scope.go:122] "RemoveContainer" containerID="8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e" Nov 02 14:28:52.235042 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:52.235021 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e\": container with ID starting with 8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e not found: ID does not exist" containerID="8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e" Nov 02 14:28:52.235094 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.235052 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e"} err="failed to get container status \"8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e\": rpc error: code = NotFound desc = could not find container \"8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e\": container with ID starting with 8ea836a0a899eeb79f02335c4ad9ddffdf3ce2f43773588eb9a8e11b51a5df7e not found: ID does not exist" Nov 02 14:28:52.281205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.281182 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47"] Nov 02 14:28:52.294139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.294115 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9ba664/payload-processing-external-model-e2e-extmod-a-9ba664-bdc6ggt47"] Nov 02 14:28:52.496492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.496418 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" podUID="5623896f-6ea6-489a-9057-466a89520539" containerName="payload-processing" containerID="cri-o://fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f" gracePeriod=25 Nov 02 14:28:52.770576 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.765753 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:52.805570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.805536 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-plugins-config-volume\") pod \"5623896f-6ea6-489a-9057-466a89520539\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " Nov 02 14:28:52.805717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.805598 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-routing-overlay-volume\") pod \"5623896f-6ea6-489a-9057-466a89520539\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " Nov 02 14:28:52.805717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.805657 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-kube-api-access-z6gm5\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-kube-api-access-z6gm5\") pod \"5623896f-6ea6-489a-9057-466a89520539\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " Nov 02 14:28:52.805885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.805870 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-provider-credentials\") pod \"5623896f-6ea6-489a-9057-466a89520539\" (UID: \"5623896f-6ea6-489a-9057-466a89520539\") " Nov 02 14:28:52.805986 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.805956 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-routing-overlay-volume" pod "5623896f-6ea6-489a-9057-466a89520539" (UID: "5623896f-6ea6-489a-9057-466a89520539"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:52.806055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.805964 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-plugins-config-volume" pod "5623896f-6ea6-489a-9057-466a89520539" (UID: "5623896f-6ea6-489a-9057-466a89520539"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:52.806291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.806273 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:52.806365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.806294 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5623896f-6ea6-489a-9057-466a89520539-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:52.808006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.807973 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-provider-credentials" pod "5623896f-6ea6-489a-9057-466a89520539" (UID: "5623896f-6ea6-489a-9057-466a89520539"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:52.808099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.808044 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-kube-api-access-z6gm5" pod "5623896f-6ea6-489a-9057-466a89520539" (UID: "5623896f-6ea6-489a-9057-466a89520539"). InnerVolumeSpecName "kube-api-access-z6gm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:52.907162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.907133 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:52.907162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:52.907162 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6gm5\" (UniqueName: \"kubernetes.io/projected/5623896f-6ea6-489a-9057-466a89520539-kube-api-access-z6gm5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:53.078298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.078218 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e918fa-af8b-47b4-bf70-576844a8b0cc" path="/var/lib/kubelet/pods/c6e918fa-af8b-47b4-bf70-576844a8b0cc/volumes" Nov 02 14:28:53.233429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.233391 4431 generic.go:356] "Generic (PLEG): container finished" podID="5623896f-6ea6-489a-9057-466a89520539" containerID="fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f" exitCode=0 Nov 02 14:28:53.233938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.233451 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" Nov 02 14:28:53.233938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.233480 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" event={"ID":"5623896f-6ea6-489a-9057-466a89520539","Type":"ContainerDied","Data":"fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f"} Nov 02 14:28:53.233938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.233545 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv" event={"ID":"5623896f-6ea6-489a-9057-466a89520539","Type":"ContainerDied","Data":"38895a3fa697626a508be5c69f2224280a3c1280aec35dee42791a8c030ee58d"} Nov 02 14:28:53.233938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.233578 4431 scope.go:122] "RemoveContainer" containerID="fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f" Nov 02 14:28:53.243784 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.243759 4431 scope.go:122] "RemoveContainer" containerID="fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f" Nov 02 14:28:53.244104 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:53.244079 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f\": container with ID starting with fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f not found: ID does not exist" containerID="fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f" Nov 02 14:28:53.244207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.244114 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f"} err="failed to get container status \"fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f\": rpc error: code = NotFound desc = could not find container \"fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f\": container with ID starting with fd31df9ace23029133495096dea6d985434d97193562d5fa2a6a0acbf1d23d3f not found: ID does not exist" Nov 02 14:28:53.297838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.297811 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv"] Nov 02 14:28:53.302507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.302478 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-12b8da/payload-processing-external-model-e2e-extmod-b-12b8da-779fntctv"] Nov 02 14:28:53.710041 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710006 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6"] Nov 02 14:28:53.710713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710698 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0ff16766-b8c1-4bab-83c5-a4732026f528" containerName="payload-processing" Nov 02 14:28:53.710765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710716 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff16766-b8c1-4bab-83c5-a4732026f528" containerName="payload-processing" Nov 02 14:28:53.710765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710737 4431 cpu_manager.go:414] "containerMap: removing container" podUID="dba85d83-ea10-41e4-8507-e9fbd2bba1a1" containerName="revoke-keys" Nov 02 14:28:53.710765 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710745 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba85d83-ea10-41e4-8507-e9fbd2bba1a1" containerName="revoke-keys" Nov 02 14:28:53.710885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710770 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5623896f-6ea6-489a-9057-466a89520539" containerName="payload-processing" Nov 02 14:28:53.710885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710792 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5623896f-6ea6-489a-9057-466a89520539" containerName="payload-processing" Nov 02 14:28:53.710885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710816 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c6e918fa-af8b-47b4-bf70-576844a8b0cc" containerName="payload-processing" Nov 02 14:28:53.710885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710823 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e918fa-af8b-47b4-bf70-576844a8b0cc" containerName="payload-processing" Nov 02 14:28:53.711018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710978 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff16766-b8c1-4bab-83c5-a4732026f528" containerName="payload-processing" Nov 02 14:28:53.711018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.710990 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba85d83-ea10-41e4-8507-e9fbd2bba1a1" containerName="revoke-keys" Nov 02 14:28:53.711018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.711002 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5623896f-6ea6-489a-9057-466a89520539" containerName="payload-processing" Nov 02 14:28:53.711018 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.711013 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e918fa-af8b-47b4-bf70-576844a8b0cc" containerName="payload-processing" Nov 02 14:28:53.714639 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.714620 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.731699 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.731675 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-plugins-e2e-extmod-a-944506\"" Nov 02 14:28:53.735409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.735385 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6"] Nov 02 14:28:53.744675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.744646 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"routing-overlay\"" Nov 02 14:28:53.744996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.744942 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-ca19ab63a2a2\"" Nov 02 14:28:53.744996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.744966 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"openshift-service-ca.crt\"" Nov 02 14:28:53.744996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.744970 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"agc-cred-8434b5bb1061\"" Nov 02 14:28:53.745241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.745007 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"e2e-cfl-622143-provider-cred\"" Nov 02 14:28:53.745241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.745028 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"payload-processing-external-model-e2e--4e96fc15-dockercfg-gffbp\"" Nov 02 14:28:53.745241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.745083 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-944506\"/\"kube-root-ca.crt\"" Nov 02 14:28:53.818615 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.818588 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4nh\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-kube-api-access-xh4nh\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.818759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.818637 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.818844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.818809 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.818844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.818832 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.926756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.925758 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4nh\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-kube-api-access-xh4nh\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.926756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.925866 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.926756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.926014 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.926756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.926058 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.926756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.926707 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.927213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.926847 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.929818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.929388 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:53.951130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:53.951110 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4nh\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-kube-api-access-xh4nh\") pod \"payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:54.028085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:54.028033 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:54.333941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:54.333908 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6"] Nov 02 14:28:54.334926 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:54.334897 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb80560a_a82b_4116_8422_d508578eba14.slice/crio-7654de1c4f131784a32ca40bab4d85f35c175f718c0a147ffbd5b9613cfdf697 WatchSource:0}: Error finding container 7654de1c4f131784a32ca40bab4d85f35c175f718c0a147ffbd5b9613cfdf697: Status 404 returned error can't find the container with id 7654de1c4f131784a32ca40bab4d85f35c175f718c0a147ffbd5b9613cfdf697 Nov 02 14:28:55.076942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.076911 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5623896f-6ea6-489a-9057-466a89520539" path="/var/lib/kubelet/pods/5623896f-6ea6-489a-9057-466a89520539/volumes" Nov 02 14:28:55.077221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.077208 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:28:55.126192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.126167 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b"] Nov 02 14:28:55.129047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.129027 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.149032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.149001 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:28:55.149174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.149160 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:28:55.151350 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.151328 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b"] Nov 02 14:28:55.242473 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.242437 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.242661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.242500 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.242764 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.242738 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwx9k\" (UniqueName: \"kubernetes.io/projected/fc48bfa2-967a-48e9-a521-f541ac888c8c-kube-api-access-bwx9k\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.251816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.251771 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" event={"ID":"bb80560a-a82b-4116-8422-d508578eba14","Type":"ContainerStarted","Data":"c59d5a6c0bff762113f44abd907e4a9983e8bf3d61e29d1af9a7312cc29f9263"} Nov 02 14:28:55.251936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.251847 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:28:55.251936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.251867 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" event={"ID":"bb80560a-a82b-4116-8422-d508578eba14","Type":"ContainerStarted","Data":"7654de1c4f131784a32ca40bab4d85f35c175f718c0a147ffbd5b9613cfdf697"} Nov 02 14:28:55.299421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.299378 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" podStartSLOduration=2.299366175 podStartE2EDuration="2.299366175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:55.295948035 +0000 UTC m=+1900.333052968" watchObservedRunningTime="2026-11-02 14:28:55.299366175 +0000 UTC m=+1900.336471107" Nov 02 14:28:55.343634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.343569 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwx9k\" (UniqueName: \"kubernetes.io/projected/fc48bfa2-967a-48e9-a521-f541ac888c8c-kube-api-access-bwx9k\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.344030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.343659 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.344030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.343698 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.344030 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:55.343876 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-8434b5bb1061: secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:55.344030 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:55.343898 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6: secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:55.344030 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:55.343945 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials podName:bb80560a-a82b-4116-8422-d508578eba14 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:55.843929711 +0000 UTC m=+1900.881034634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" (UID: "bb80560a-a82b-4116-8422-d508578eba14") : secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:55.344926 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.344905 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.346592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.346571 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.385500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.385462 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwx9k\" (UniqueName: \"kubernetes.io/projected/fc48bfa2-967a-48e9-a521-f541ac888c8c-kube-api-access-bwx9k\") pod \"maas-api-revoke-sub-753e7808-mvn9b\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.444166 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.444136 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:55.510214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.510167 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6"] Nov 02 14:28:55.540512 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.540481 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw"] Nov 02 14:28:55.544663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.544641 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.577825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.576716 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw"] Nov 02 14:28:55.647234 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.647181 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.647401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.647340 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.647457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.647427 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47mr\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-kube-api-access-h47mr\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.647542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.647516 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.748581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.748540 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.748943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.748919 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.749091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.749075 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47mr\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-kube-api-access-h47mr\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.749223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.749210 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.750378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.750355 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.758795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.756356 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.758795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.756801 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.782534 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.782498 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47mr\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-kube-api-access-h47mr\") pod \"payload-processing-external-model-e2e-extmod-a-944506-b896rhknw\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:55.813146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.813063 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b"] Nov 02 14:28:55.850467 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:55.850441 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-8434b5bb1061: secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:55.850605 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:55.850477 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6: secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:55.850605 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:55.850528 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials podName:bb80560a-a82b-4116-8422-d508578eba14 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:56.850510467 +0000 UTC m=+1901.887615381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" (UID: "bb80560a-a82b-4116-8422-d508578eba14") : secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:55.864004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:55.863981 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:56.245124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.245096 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw"] Nov 02 14:28:56.245582 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:56.245552 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d4baef_4ab0_4e3f_8e07_802f0a7d1c84.slice/crio-352b23c7d0487ffba3b2d565cd57bafcc6240bd9fcd57e1816d6a630d376a9b2 WatchSource:0}: Error finding container 352b23c7d0487ffba3b2d565cd57bafcc6240bd9fcd57e1816d6a630d376a9b2: Status 404 returned error can't find the container with id 352b23c7d0487ffba3b2d565cd57bafcc6240bd9fcd57e1816d6a630d376a9b2 Nov 02 14:28:56.262225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.262195 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" event={"ID":"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84","Type":"ContainerStarted","Data":"352b23c7d0487ffba3b2d565cd57bafcc6240bd9fcd57e1816d6a630d376a9b2"} Nov 02 14:28:56.265312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.265282 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" event={"ID":"fc48bfa2-967a-48e9-a521-f541ac888c8c","Type":"ContainerStarted","Data":"aadafb47bc2a1c9a6861cd982d18b6a6af6aed264c808af07159ba6aab57ad21"} Nov 02 14:28:56.296244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.296221 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn"] Nov 02 14:28:56.299276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.299257 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.367801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.362860 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn"] Nov 02 14:28:56.367801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.366350 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d0c9b9-openshift-default-dockercfg-lchmd\"" Nov 02 14:28:56.368349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.368062 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-data\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.368349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.368115 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-envoy\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.368349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.368312 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-podinfo\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.368515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.368367 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-certs\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.368515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.368420 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mc8c\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-kube-api-access-8mc8c\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.375799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.369818 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istiod-ca-cert\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.375799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.370590 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-socket\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.375799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.370769 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-credential-socket\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.375799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.370877 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-token\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.472772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.472679 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istiod-ca-cert\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.472772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.472747 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-socket\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.472840 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-credential-socket\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.472869 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-token\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.473009 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-data\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.473043 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-envoy\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.473123 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-podinfo\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.473171 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-certs\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.473182 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-socket\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.473223 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mc8c\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-kube-api-access-8mc8c\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.473410 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-data\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.473661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.473637 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-credential-socket\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.474147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.474124 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-certs\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.474748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.474722 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istiod-ca-cert\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.476071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.476052 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-envoy\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.476450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.476430 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-podinfo\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.503896 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.503861 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mc8c\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-kube-api-access-8mc8c\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.520795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.518148 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-token\") pod \"e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.631801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.629049 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:56.882386 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:56.881195 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-8434b5bb1061: secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:56.882386 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:56.881234 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6: secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:56.882386 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:56.881307 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials podName:bb80560a-a82b-4116-8422-d508578eba14 nodeName:}" failed. No retries permitted until 2026-11-02 14:28:58.881288177 +0000 UTC m=+1903.918393103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" (UID: "bb80560a-a82b-4116-8422-d508578eba14") : secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:56.882386 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.881595 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w"] Nov 02 14:28:56.928422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.928030 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:56.951314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.949381 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f72bc6\"/\"premium-w1-f72bc6-kserve-self-signed-certs\"" Nov 02 14:28:56.958489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.958460 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w"] Nov 02 14:28:56.998644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.998609 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537d9533-ce0c-4f93-be2f-684bce472885-tls-certs\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:56.999582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.999557 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-tmp-dir\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:56.999753 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.999732 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fht9r\" (UniqueName: \"kubernetes.io/projected/537d9533-ce0c-4f93-be2f-684bce472885-kube-api-access-fht9r\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:56.999945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:56.999924 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-dshm\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.000098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.000081 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-model-cache\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.000260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.000242 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-home\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.102567 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-tmp-dir\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.102613 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fht9r\" (UniqueName: \"kubernetes.io/projected/537d9533-ce0c-4f93-be2f-684bce472885-kube-api-access-fht9r\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.102671 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-dshm\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.102708 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-model-cache\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.102766 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-home\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.102878 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537d9533-ce0c-4f93-be2f-684bce472885-tls-certs\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.104213 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-model-cache\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.104763 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-home\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.108799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.105176 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-tmp-dir\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.163871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.134058 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-dshm\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.163871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.143499 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fht9r\" (UniqueName: \"kubernetes.io/projected/537d9533-ce0c-4f93-be2f-684bce472885-kube-api-access-fht9r\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.163871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.144199 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537d9533-ce0c-4f93-be2f-684bce472885-tls-certs\") pod \"premium-w1-f72bc6-kserve-7bd57c9675-wls2w\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.294361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.293629 4431 generic.go:356] "Generic (PLEG): container finished" podID="fc48bfa2-967a-48e9-a521-f541ac888c8c" containerID="e8703b5a15baabd7c8f2126a547eca7ff91397709513ae1887d3dc2c80a23836" exitCode=0 Nov 02 14:28:57.295027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.294900 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" event={"ID":"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84","Type":"ContainerStarted","Data":"e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c"} Nov 02 14:28:57.295027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.294960 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:28:57.295027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.294985 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" event={"ID":"fc48bfa2-967a-48e9-a521-f541ac888c8c","Type":"ContainerDied","Data":"e8703b5a15baabd7c8f2126a547eca7ff91397709513ae1887d3dc2c80a23836"} Nov 02 14:28:57.326263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.323759 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:57.370372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.340306 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" podStartSLOduration=2.340290939 podStartE2EDuration="2.340290939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:57.338482512 +0000 UTC m=+1902.375587457" watchObservedRunningTime="2026-11-02 14:28:57.340290939 +0000 UTC m=+1902.377395874" Nov 02 14:28:57.665162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.665133 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn"] Nov 02 14:28:57.667442 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:28:57.667374 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2e3387_dee2_46b2_ad82_fcd52485a3aa.slice/crio-8f196a995db3b7ec7fa52af8373afff4fbedc0204aacf66c7245defb2f47dadd WatchSource:0}: Error finding container 8f196a995db3b7ec7fa52af8373afff4fbedc0204aacf66c7245defb2f47dadd: Status 404 returned error can't find the container with id 8f196a995db3b7ec7fa52af8373afff4fbedc0204aacf66c7245defb2f47dadd Nov 02 14:28:57.671865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.670656 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:28:57.671865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.670735 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:28:57.671865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.670796 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:28:57.784909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:57.783131 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w"] Nov 02 14:28:58.322800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.322578 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" event={"ID":"fe2e3387-dee2-46b2-ad82-fcd52485a3aa","Type":"ContainerStarted","Data":"2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70"} Nov 02 14:28:58.322800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.322736 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" event={"ID":"fe2e3387-dee2-46b2-ad82-fcd52485a3aa","Type":"ContainerStarted","Data":"8f196a995db3b7ec7fa52af8373afff4fbedc0204aacf66c7245defb2f47dadd"} Nov 02 14:28:58.340699 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.340668 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" event={"ID":"537d9533-ce0c-4f93-be2f-684bce472885","Type":"ContainerStarted","Data":"41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab"} Nov 02 14:28:58.340930 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.340908 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" event={"ID":"537d9533-ce0c-4f93-be2f-684bce472885","Type":"ContainerStarted","Data":"09a38aee47073c0d4473e75e4c15ad726f7cecb0bd163ab12c6b35dc84f88e04"} Nov 02 14:28:58.500803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.499755 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" podStartSLOduration=2.499733657 podStartE2EDuration="2.499733657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:58.42359456 +0000 UTC m=+1903.460699485" watchObservedRunningTime="2026-11-02 14:28:58.499733657 +0000 UTC m=+1903.536838590" Nov 02 14:28:58.515890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.515828 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" podStartSLOduration=2.515809155 podStartE2EDuration="2.515809155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:58.51502226 +0000 UTC m=+1903.552127194" watchObservedRunningTime="2026-11-02 14:28:58.515809155 +0000 UTC m=+1903.552914090" Nov 02 14:28:58.633472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.630169 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:28:58.633755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.633729 4431 patch_prober.go:29] interesting pod/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:28:58.634017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:58.633986 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:28:58.989765 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:58.983014 4431 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-944506/agc-cred-8434b5bb1061: secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:58.990873 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:58.990853 4431 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6: secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:58.991103 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:28:58.991089 4431 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials podName:bb80560a-a82b-4116-8422-d508578eba14 nodeName:}" failed. No retries permitted until 2026-11-02 14:29:02.991062482 +0000 UTC m=+1908.028167415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" (UID: "bb80560a-a82b-4116-8422-d508578eba14") : secret "agc-cred-8434b5bb1061" not found Nov 02 14:28:59.231817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.229279 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:59.246998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.246968 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" podUID="17b062be-55b8-4b08-8512-e8c86b10e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:28:59.266674 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.259836 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:28:59.328369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.328326 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:28:59.413275 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.413237 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-openshift-ca\") pod \"fc48bfa2-967a-48e9-a521-f541ac888c8c\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " Nov 02 14:28:59.414142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.414120 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc48bfa2-967a-48e9-a521-f541ac888c8c-kube-api-access-bwx9k\" (UniqueName: \"kubernetes.io/projected/fc48bfa2-967a-48e9-a521-f541ac888c8c-kube-api-access-bwx9k\") pod \"fc48bfa2-967a-48e9-a521-f541ac888c8c\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " Nov 02 14:28:59.414847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.414825 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-xks-ca\") pod \"fc48bfa2-967a-48e9-a521-f541ac888c8c\" (UID: \"fc48bfa2-967a-48e9-a521-f541ac888c8c\") " Nov 02 14:28:59.415880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.413891 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-openshift-ca" pod "fc48bfa2-967a-48e9-a521-f541ac888c8c" (UID: "fc48bfa2-967a-48e9-a521-f541ac888c8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:28:59.515865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.507665 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" Nov 02 14:28:59.515865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.507847 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b" event={"ID":"fc48bfa2-967a-48e9-a521-f541ac888c8c","Type":"ContainerDied","Data":"aadafb47bc2a1c9a6861cd982d18b6a6af6aed264c808af07159ba6aab57ad21"} Nov 02 14:28:59.515865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.507890 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aadafb47bc2a1c9a6861cd982d18b6a6af6aed264c808af07159ba6aab57ad21" Nov 02 14:28:59.516949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.516918 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:59.519430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.519402 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-xks-ca" pod "fc48bfa2-967a-48e9-a521-f541ac888c8c" (UID: "fc48bfa2-967a-48e9-a521-f541ac888c8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:28:59.542999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.539922 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc48bfa2-967a-48e9-a521-f541ac888c8c-kube-api-access-bwx9k" pod "fc48bfa2-967a-48e9-a521-f541ac888c8c" (UID: "fc48bfa2-967a-48e9-a521-f541ac888c8c"). InnerVolumeSpecName "kube-api-access-bwx9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:28:59.619940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.619898 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc48bfa2-967a-48e9-a521-f541ac888c8c-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:59.620171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.620154 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwx9k\" (UniqueName: \"kubernetes.io/projected/fc48bfa2-967a-48e9-a521-f541ac888c8c-kube-api-access-bwx9k\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:28:59.637082 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:28:59.637051 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:29:00.535800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:00.527906 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:29:00.535800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:00.534033 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:29:02.294932 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.294883 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" podUID="bb80560a-a82b-4116-8422-d508578eba14" containerName="payload-processing" containerID="cri-o://c59d5a6c0bff762113f44abd907e4a9983e8bf3d61e29d1af9a7312cc29f9263" gracePeriod=25 Nov 02 14:29:02.542849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.542819 4431 generic.go:356] "Generic (PLEG): container finished" podID="bb80560a-a82b-4116-8422-d508578eba14" containerID="c59d5a6c0bff762113f44abd907e4a9983e8bf3d61e29d1af9a7312cc29f9263" exitCode=0 Nov 02 14:29:02.542990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.542889 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" event={"ID":"bb80560a-a82b-4116-8422-d508578eba14","Type":"ContainerDied","Data":"c59d5a6c0bff762113f44abd907e4a9983e8bf3d61e29d1af9a7312cc29f9263"} Nov 02 14:29:02.542990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.542941 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" event={"ID":"bb80560a-a82b-4116-8422-d508578eba14","Type":"ContainerDied","Data":"7654de1c4f131784a32ca40bab4d85f35c175f718c0a147ffbd5b9613cfdf697"} Nov 02 14:29:02.542990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.542967 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7654de1c4f131784a32ca40bab4d85f35c175f718c0a147ffbd5b9613cfdf697" Nov 02 14:29:02.551622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.551568 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:29:02.601376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.600592 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials\") pod \"bb80560a-a82b-4116-8422-d508578eba14\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " Nov 02 14:29:02.601376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.600792 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-routing-overlay-volume\") pod \"bb80560a-a82b-4116-8422-d508578eba14\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " Nov 02 14:29:02.601376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.601235 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-kube-api-access-xh4nh\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-kube-api-access-xh4nh\") pod \"bb80560a-a82b-4116-8422-d508578eba14\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " Nov 02 14:29:02.601376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.601301 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-plugins-config-volume\") pod \"bb80560a-a82b-4116-8422-d508578eba14\" (UID: \"bb80560a-a82b-4116-8422-d508578eba14\") " Nov 02 14:29:02.601376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.601299 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-routing-overlay-volume" pod "bb80560a-a82b-4116-8422-d508578eba14" (UID: "bb80560a-a82b-4116-8422-d508578eba14"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:02.603607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.601751 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-plugins-config-volume" pod "bb80560a-a82b-4116-8422-d508578eba14" (UID: "bb80560a-a82b-4116-8422-d508578eba14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:02.603607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.602035 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:02.603607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.602056 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb80560a-a82b-4116-8422-d508578eba14-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:02.604552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.604088 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials" pod "bb80560a-a82b-4116-8422-d508578eba14" (UID: "bb80560a-a82b-4116-8422-d508578eba14"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:02.604552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.604237 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-kube-api-access-xh4nh" pod "bb80560a-a82b-4116-8422-d508578eba14" (UID: "bb80560a-a82b-4116-8422-d508578eba14"). InnerVolumeSpecName "kube-api-access-xh4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:02.703370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.703336 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh4nh\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-kube-api-access-xh4nh\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:02.703370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:02.703366 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb80560a-a82b-4116-8422-d508578eba14-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:03.550541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:03.550449 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6" Nov 02 14:29:03.616595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:03.616560 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6"] Nov 02 14:29:03.629771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:03.629739 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-8578qcnx6"] Nov 02 14:29:05.076977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:05.076934 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb80560a-a82b-4116-8422-d508578eba14" path="/var/lib/kubelet/pods/bb80560a-a82b-4116-8422-d508578eba14/volumes" Nov 02 14:29:07.326796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:07.326742 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:29:07.339355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:07.339334 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:29:07.588135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:07.588062 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:29:08.185128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.185092 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm"] Nov 02 14:29:08.185941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.185915 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fc48bfa2-967a-48e9-a521-f541ac888c8c" containerName="revoke-keys" Nov 02 14:29:08.185941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.185940 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc48bfa2-967a-48e9-a521-f541ac888c8c" containerName="revoke-keys" Nov 02 14:29:08.186124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.185961 4431 cpu_manager.go:414] "containerMap: removing container" podUID="bb80560a-a82b-4116-8422-d508578eba14" containerName="payload-processing" Nov 02 14:29:08.186124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.185969 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb80560a-a82b-4116-8422-d508578eba14" containerName="payload-processing" Nov 02 14:29:08.186215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.186157 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc48bfa2-967a-48e9-a521-f541ac888c8c" containerName="revoke-keys" Nov 02 14:29:08.186215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.186171 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb80560a-a82b-4116-8422-d508578eba14" containerName="payload-processing" Nov 02 14:29:08.189761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.189738 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.207672 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.207653 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm"] Nov 02 14:29:08.216713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.216691 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:29:08.216869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.216737 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:29:08.220231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.220208 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw"] Nov 02 14:29:08.223041 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.223021 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:29:08.231108 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.231069 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.231248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.231207 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.231321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.231255 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vg4q\" (UniqueName: \"kubernetes.io/projected/ef4ef40c-226c-4edc-b775-d71291e4ca90-kube-api-access-6vg4q\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.332424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.332395 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.332873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.332462 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.332873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.332486 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vg4q\" (UniqueName: \"kubernetes.io/projected/ef4ef40c-226c-4edc-b775-d71291e4ca90-kube-api-access-6vg4q\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.333070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.333047 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.335025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.335001 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.356903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.356882 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vg4q\" (UniqueName: \"kubernetes.io/projected/ef4ef40c-226c-4edc-b775-d71291e4ca90-kube-api-access-6vg4q\") pod \"maas-api-revoke-sub-25361b0a-44jfm\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.503079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.503022 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:08.806214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:08.806189 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm"] Nov 02 14:29:08.807092 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:29:08.807069 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4ef40c_226c_4edc_b775_d71291e4ca90.slice/crio-7914435e1a3880251cd9092850045e4efdd1aad879cde58c09bc0985f0a04afa WatchSource:0}: Error finding container 7914435e1a3880251cd9092850045e4efdd1aad879cde58c09bc0985f0a04afa: Status 404 returned error can't find the container with id 7914435e1a3880251cd9092850045e4efdd1aad879cde58c09bc0985f0a04afa Nov 02 14:29:09.596426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:09.596388 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" event={"ID":"ef4ef40c-226c-4edc-b775-d71291e4ca90","Type":"ContainerStarted","Data":"69d05f9110bce0c52a6ed4cad913a9c50d28974e52a9726ec88495d516802342"} Nov 02 14:29:09.596941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:09.596440 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" event={"ID":"ef4ef40c-226c-4edc-b775-d71291e4ca90","Type":"ContainerStarted","Data":"7914435e1a3880251cd9092850045e4efdd1aad879cde58c09bc0985f0a04afa"} Nov 02 14:29:09.640279 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:09.640219 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" podStartSLOduration=1.640206956 podStartE2EDuration="1.640206956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:09.639140201 +0000 UTC m=+1914.676245132" watchObservedRunningTime="2026-11-02 14:29:09.640206956 +0000 UTC m=+1914.677311958" Nov 02 14:29:10.604702 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:10.604662 4431 generic.go:356] "Generic (PLEG): container finished" podID="ef4ef40c-226c-4edc-b775-d71291e4ca90" containerID="69d05f9110bce0c52a6ed4cad913a9c50d28974e52a9726ec88495d516802342" exitCode=0 Nov 02 14:29:10.605136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:10.604752 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" event={"ID":"ef4ef40c-226c-4edc-b775-d71291e4ca90","Type":"ContainerDied","Data":"69d05f9110bce0c52a6ed4cad913a9c50d28974e52a9726ec88495d516802342"} Nov 02 14:29:11.772172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.772146 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:11.880447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.880374 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4ef40c-226c-4edc-b775-d71291e4ca90-kube-api-access-6vg4q\" (UniqueName: \"kubernetes.io/projected/ef4ef40c-226c-4edc-b775-d71291e4ca90-kube-api-access-6vg4q\") pod \"ef4ef40c-226c-4edc-b775-d71291e4ca90\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " Nov 02 14:29:11.880593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.880490 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-xks-ca\") pod \"ef4ef40c-226c-4edc-b775-d71291e4ca90\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " Nov 02 14:29:11.880593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.880530 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-openshift-ca\") pod \"ef4ef40c-226c-4edc-b775-d71291e4ca90\" (UID: \"ef4ef40c-226c-4edc-b775-d71291e4ca90\") " Nov 02 14:29:11.880990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.880960 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-openshift-ca" pod "ef4ef40c-226c-4edc-b775-d71291e4ca90" (UID: "ef4ef40c-226c-4edc-b775-d71291e4ca90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:11.882395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.882375 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4ef40c-226c-4edc-b775-d71291e4ca90-kube-api-access-6vg4q" pod "ef4ef40c-226c-4edc-b775-d71291e4ca90" (UID: "ef4ef40c-226c-4edc-b775-d71291e4ca90"). InnerVolumeSpecName "kube-api-access-6vg4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:11.882537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.882522 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-xks-ca" pod "ef4ef40c-226c-4edc-b775-d71291e4ca90" (UID: "ef4ef40c-226c-4edc-b775-d71291e4ca90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:29:11.982417 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.982393 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:11.982514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.982418 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4ef40c-226c-4edc-b775-d71291e4ca90-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:11.982514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:11.982429 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vg4q\" (UniqueName: \"kubernetes.io/projected/ef4ef40c-226c-4edc-b775-d71291e4ca90-kube-api-access-6vg4q\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:12.619227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:12.619197 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" Nov 02 14:29:12.619227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:12.619206 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm" event={"ID":"ef4ef40c-226c-4edc-b775-d71291e4ca90","Type":"ContainerDied","Data":"7914435e1a3880251cd9092850045e4efdd1aad879cde58c09bc0985f0a04afa"} Nov 02 14:29:12.619433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:12.619257 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7914435e1a3880251cd9092850045e4efdd1aad879cde58c09bc0985f0a04afa" Nov 02 14:29:13.221131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.221077 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" podUID="b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" containerName="payload-processing" containerID="cri-o://e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c" gracePeriod=25 Nov 02 14:29:13.507708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.507668 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:29:13.607530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.607501 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-routing-overlay-volume\") pod \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " Nov 02 14:29:13.607675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.607535 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-provider-credentials\") pod \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " Nov 02 14:29:13.607675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.607562 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-plugins-config-volume\") pod \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " Nov 02 14:29:13.607797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.607760 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-kube-api-access-h47mr\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-kube-api-access-h47mr\") pod \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\" (UID: \"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84\") " Nov 02 14:29:13.608006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.607977 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-routing-overlay-volume" pod "b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" (UID: "b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:13.608100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.608014 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-plugins-config-volume" pod "b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" (UID: "b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:29:13.608285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.608267 4431 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-routing-overlay-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:13.608326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.608288 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:13.609912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.609886 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-kube-api-access-h47mr" pod "b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" (UID: "b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84"). InnerVolumeSpecName "kube-api-access-h47mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:13.610010 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.609921 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-provider-credentials" pod "b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" (UID: "b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:13.627392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.627367 4431 generic.go:356] "Generic (PLEG): container finished" podID="b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" containerID="e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c" exitCode=0 Nov 02 14:29:13.627511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.627423 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" Nov 02 14:29:13.627511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.627444 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" event={"ID":"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84","Type":"ContainerDied","Data":"e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c"} Nov 02 14:29:13.627511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.627496 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw" event={"ID":"b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84","Type":"ContainerDied","Data":"352b23c7d0487ffba3b2d565cd57bafcc6240bd9fcd57e1816d6a630d376a9b2"} Nov 02 14:29:13.627640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.627518 4431 scope.go:122] "RemoveContainer" containerID="e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c" Nov 02 14:29:13.640030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.640014 4431 scope.go:122] "RemoveContainer" containerID="e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c" Nov 02 14:29:13.640307 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:29:13.640285 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c\": container with ID starting with e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c not found: ID does not exist" containerID="e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c" Nov 02 14:29:13.640386 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.640314 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c"} err="failed to get container status \"e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c\": rpc error: code = NotFound desc = could not find container \"e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c\": container with ID starting with e1eb1d99a597b818addcd4bc2b189f420f55573fc23b63389b62db616146e06c not found: ID does not exist" Nov 02 14:29:13.687150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.687123 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw"] Nov 02 14:29:13.696211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.696182 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-944506/payload-processing-external-model-e2e-extmod-a-944506-b896rhknw"] Nov 02 14:29:13.709907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.709885 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47mr\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-kube-api-access-h47mr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:13.709907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:13.709904 4431 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84-provider-credentials\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:15.076841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:15.076809 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" path="/var/lib/kubelet/pods/b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84/volumes" Nov 02 14:29:17.308801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:17.308033 4431 scope.go:122] "RemoveContainer" containerID="c5c0a59ffb988f42f30b3ad4aa5b53617ec95805dc4333e5a827123caba3abcc" Nov 02 14:29:18.756032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:18.755864 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-97tgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:29:18.756032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:18.755952 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:29:19.155800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.145324 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" podUID="17b062be-55b8-4b08-8512-e8c86b10e317" containerName="main" probeResult="failure" output="Get \"https://10.133.0.188:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:29:19.291798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.290905 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:29:19.300797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.299897 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:29:19.423854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.422837 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 02 14:29:19.521803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.517926 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 14:29:19.603802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.601857 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" podUID="537d9533-ce0c-4f93-be2f-684bce472885" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:29:19.911800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.911378 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v"] Nov 02 14:29:19.935158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.935129 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" containerName="payload-processing" Nov 02 14:29:19.935354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.935335 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" containerName="payload-processing" Nov 02 14:29:19.935496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.935483 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ef4ef40c-226c-4edc-b775-d71291e4ca90" containerName="revoke-keys" Nov 02 14:29:19.935592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.935580 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4ef40c-226c-4edc-b775-d71291e4ca90" containerName="revoke-keys" Nov 02 14:29:19.935929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.935909 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d4baef-4ab0-4e3f-8e07-802f0a7d1c84" containerName="payload-processing" Nov 02 14:29:19.936095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.936079 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4ef40c-226c-4edc-b775-d71291e4ca90" containerName="revoke-keys" Nov 02 14:29:19.945946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.945924 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v"] Nov 02 14:29:19.946212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.946166 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:19.978415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.975256 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5a4525\"/\"explicit-model-5a4525-kserve-self-signed-certs\"" Nov 02 14:29:19.981798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.978717 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5a4525\"/\"default-dockercfg-7scz7\"" Nov 02 14:29:19.985796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.983853 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5a4525\"/\"openshift-service-ca.crt\"" Nov 02 14:29:19.985796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.983995 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5a4525\"/\"kube-root-ca.crt\"" Nov 02 14:29:19.994812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.991560 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qp4\" (UniqueName: \"kubernetes.io/projected/ead48bca-6f9f-4201-b96b-237e4f2a178c-kube-api-access-74qp4\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:19.994812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.991608 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-home\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:19.994812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.991742 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-model-cache\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:19.994812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.991807 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-dshm\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:19.994812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.991961 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead48bca-6f9f-4201-b96b-237e4f2a178c-tls-certs\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:19.994812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:19.992046 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-tmp-dir\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.094875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.093530 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-dshm\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.094875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.093642 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead48bca-6f9f-4201-b96b-237e4f2a178c-tls-certs\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.094875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.093725 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-tmp-dir\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.094875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.093812 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qp4\" (UniqueName: \"kubernetes.io/projected/ead48bca-6f9f-4201-b96b-237e4f2a178c-kube-api-access-74qp4\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.094875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.093847 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-home\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.094875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.094008 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-model-cache\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.094875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.094500 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-model-cache\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.095404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.095122 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-home\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.096041 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.095981 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-tmp-dir\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.104850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.101161 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-dshm\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.104850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.101539 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead48bca-6f9f-4201-b96b-237e4f2a178c-tls-certs\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.150800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.134573 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qp4\" (UniqueName: \"kubernetes.io/projected/ead48bca-6f9f-4201-b96b-237e4f2a178c-kube-api-access-74qp4\") pod \"explicit-model-5a4525-kserve-597b9fffb7-9ww7v\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.330451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.330362 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:20.697797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.688582 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v"] Nov 02 14:29:20.751333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:20.748551 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" event={"ID":"ead48bca-6f9f-4201-b96b-237e4f2a178c","Type":"ContainerStarted","Data":"decdd6b7655dde8514f9b6cf2d12cab381020c65dd95774539f7d09e54fb884e"} Nov 02 14:29:21.305863 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:29:21.305752 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:29:21.759259 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:21.759225 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" event={"ID":"ead48bca-6f9f-4201-b96b-237e4f2a178c","Type":"ContainerStarted","Data":"95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1"} Nov 02 14:29:21.827043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:21.826977 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" podStartSLOduration=2.826959315 podStartE2EDuration="2.826959315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:21.823203907 +0000 UTC m=+1926.860308841" watchObservedRunningTime="2026-11-02 14:29:21.826959315 +0000 UTC m=+1926.864064250" Nov 02 14:29:22.331498 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:22.331466 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:25.229462 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:25.229426 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v"] Nov 02 14:29:40.253744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.253656 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" podUID="ead48bca-6f9f-4201-b96b-237e4f2a178c" containerName="main" containerID="cri-o://95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1" gracePeriod=45 Nov 02 14:29:40.555459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.555435 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:40.674752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.674723 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead48bca-6f9f-4201-b96b-237e4f2a178c-kube-api-access-74qp4\" (UniqueName: \"kubernetes.io/projected/ead48bca-6f9f-4201-b96b-237e4f2a178c-kube-api-access-74qp4\") pod \"ead48bca-6f9f-4201-b96b-237e4f2a178c\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " Nov 02 14:29:40.674925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.674865 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-model-cache\") pod \"ead48bca-6f9f-4201-b96b-237e4f2a178c\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " Nov 02 14:29:40.674925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.674888 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead48bca-6f9f-4201-b96b-237e4f2a178c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ead48bca-6f9f-4201-b96b-237e4f2a178c-tls-certs\") pod \"ead48bca-6f9f-4201-b96b-237e4f2a178c\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " Nov 02 14:29:40.674925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.674918 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-home\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-home\") pod \"ead48bca-6f9f-4201-b96b-237e4f2a178c\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " Nov 02 14:29:40.675100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.674968 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-dshm\") pod \"ead48bca-6f9f-4201-b96b-237e4f2a178c\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " Nov 02 14:29:40.675100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.675027 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-tmp-dir\") pod \"ead48bca-6f9f-4201-b96b-237e4f2a178c\" (UID: \"ead48bca-6f9f-4201-b96b-237e4f2a178c\") " Nov 02 14:29:40.675209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.675148 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-model-cache" pod "ead48bca-6f9f-4201-b96b-237e4f2a178c" (UID: "ead48bca-6f9f-4201-b96b-237e4f2a178c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:29:40.675271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.675237 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-home" pod "ead48bca-6f9f-4201-b96b-237e4f2a178c" (UID: "ead48bca-6f9f-4201-b96b-237e4f2a178c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:29:40.675405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.675381 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-tmp-dir" pod "ead48bca-6f9f-4201-b96b-237e4f2a178c" (UID: "ead48bca-6f9f-4201-b96b-237e4f2a178c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:29:40.675495 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.675474 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:40.675568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.675497 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:40.675568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.675511 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:40.677119 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.677090 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead48bca-6f9f-4201-b96b-237e4f2a178c-kube-api-access-74qp4" pod "ead48bca-6f9f-4201-b96b-237e4f2a178c" (UID: "ead48bca-6f9f-4201-b96b-237e4f2a178c"). InnerVolumeSpecName "kube-api-access-74qp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:29:40.677243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.677119 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead48bca-6f9f-4201-b96b-237e4f2a178c-tls-certs" pod "ead48bca-6f9f-4201-b96b-237e4f2a178c" (UID: "ead48bca-6f9f-4201-b96b-237e4f2a178c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:29:40.677311 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.677269 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-dshm" pod "ead48bca-6f9f-4201-b96b-237e4f2a178c" (UID: "ead48bca-6f9f-4201-b96b-237e4f2a178c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:29:40.776298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.776274 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead48bca-6f9f-4201-b96b-237e4f2a178c-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:40.776298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.776296 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead48bca-6f9f-4201-b96b-237e4f2a178c-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:40.776447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.776307 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qp4\" (UniqueName: \"kubernetes.io/projected/ead48bca-6f9f-4201-b96b-237e4f2a178c-kube-api-access-74qp4\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:29:40.899122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.899095 4431 generic.go:356] "Generic (PLEG): container finished" podID="ead48bca-6f9f-4201-b96b-237e4f2a178c" containerID="95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1" exitCode=0 Nov 02 14:29:40.899264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.899137 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" Nov 02 14:29:40.899264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.899177 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" event={"ID":"ead48bca-6f9f-4201-b96b-237e4f2a178c","Type":"ContainerDied","Data":"95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1"} Nov 02 14:29:40.899264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.899222 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v" event={"ID":"ead48bca-6f9f-4201-b96b-237e4f2a178c","Type":"ContainerDied","Data":"decdd6b7655dde8514f9b6cf2d12cab381020c65dd95774539f7d09e54fb884e"} Nov 02 14:29:40.899264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.899244 4431 scope.go:122] "RemoveContainer" containerID="95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1" Nov 02 14:29:40.909241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.909223 4431 scope.go:122] "RemoveContainer" containerID="95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1" Nov 02 14:29:40.909466 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:29:40.909447 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1\": container with ID starting with 95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1 not found: ID does not exist" containerID="95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1" Nov 02 14:29:40.909529 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.909473 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1"} err="failed to get container status \"95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1\": rpc error: code = NotFound desc = could not find container \"95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1\": container with ID starting with 95a7fb3562d1e7253cbdcbfc0cd5186e82f61b217c9d85e41648e102e16157f1 not found: ID does not exist" Nov 02 14:29:40.955633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.955606 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v"] Nov 02 14:29:40.968974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:40.968954 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5a4525/explicit-model-5a4525-kserve-597b9fffb7-9ww7v"] Nov 02 14:29:41.077421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:41.077398 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead48bca-6f9f-4201-b96b-237e4f2a178c" path="/var/lib/kubelet/pods/ead48bca-6f9f-4201-b96b-237e4f2a178c/volumes" Nov 02 14:29:43.994885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:43.994845 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:29:51.284756 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:29:51.284714 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:29:53.994557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:53.994531 4431 generic.go:356] "Generic (PLEG): container finished" podID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerID="2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70" exitCode=0 Nov 02 14:29:53.994931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:53.994601 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" event={"ID":"fe2e3387-dee2-46b2-ad82-fcd52485a3aa","Type":"ContainerDied","Data":"2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70"} Nov 02 14:29:53.995005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:53.994991 4431 scope.go:122] "RemoveContainer" containerID="2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70" Nov 02 14:29:53.996408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:53.996381 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:29:53.996477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:53.996451 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:29:53.996514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:53.996482 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:29:55.002533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:55.002500 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" event={"ID":"fe2e3387-dee2-46b2-ad82-fcd52485a3aa","Type":"ContainerStarted","Data":"1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c"} Nov 02 14:29:55.630414 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:55.630381 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:29:55.631617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:55.631589 4431 patch_prober.go:29] interesting pod/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:29:55.631767 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:55.631637 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:29:56.630175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:56.630136 4431 patch_prober.go:29] interesting pod/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:29:56.630595 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:56.630206 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:29:57.630654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:57.630619 4431 patch_prober.go:29] interesting pod/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:29:57.631181 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:57.630677 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:29:58.630252 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:58.630219 4431 patch_prober.go:29] interesting pod/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:29:58.630424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:58.630281 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:29:59.630472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:59.630439 4431 patch_prober.go:29] interesting pod/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 02 14:29:59.630906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:29:59.630502 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 02 14:30:00.161947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.161912 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6"] Nov 02 14:30:00.162706 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.162679 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ead48bca-6f9f-4201-b96b-237e4f2a178c" containerName="main" Nov 02 14:30:00.162706 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.162701 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead48bca-6f9f-4201-b96b-237e4f2a178c" containerName="main" Nov 02 14:30:00.162890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.162817 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead48bca-6f9f-4201-b96b-237e4f2a178c" containerName="main" Nov 02 14:30:00.165216 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.165197 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:00.165900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.165882 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx"] Nov 02 14:30:00.169232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.169210 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:00.169831 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.169812 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4"] Nov 02 14:30:00.172855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.172836 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:00.174423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.174405 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff"] Nov 02 14:30:00.199675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.197008 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:00.200208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.200087 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m"] Nov 02 14:30:00.201509 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.201481 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:30:00.203848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.203467 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:00.206157 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.206097 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4"] Nov 02 14:30:00.223996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.222805 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6"] Nov 02 14:30:00.233178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.233152 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx"] Nov 02 14:30:00.237763 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.237573 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff"] Nov 02 14:30:00.244412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.244386 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2sx\" (UniqueName: \"kubernetes.io/projected/54b19435-f528-48b4-b807-70e9c6c80abf-kube-api-access-pt2sx\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6\" (UID: \"54b19435-f528-48b4-b807-70e9c6c80abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:00.244518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.244462 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p8wr\" (UniqueName: \"kubernetes.io/projected/c1aea439-9e6a-42dd-a70f-e30d47a19f19-kube-api-access-8p8wr\") pod \"maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4\" (UID: \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:00.244611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.244584 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ef1614-c978-4046-85fe-15c48736d0da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx\" (UID: \"76ef1614-c978-4046-85fe-15c48736d0da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:00.244728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.244667 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4z7\" (UniqueName: \"kubernetes.io/projected/ce56cca9-81f2-433f-baf5-a61fb490f1f5-kube-api-access-2q4z7\") pod \"maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff\" (UID: \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:00.244850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.244824 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b19435-f528-48b4-b807-70e9c6c80abf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6\" (UID: \"54b19435-f528-48b4-b807-70e9c6c80abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:00.244953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.244915 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce56cca9-81f2-433f-baf5-a61fb490f1f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff\" (UID: \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:00.245190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.245114 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7316d0-07ef-4664-9345-d00cd333551d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m\" (UID: \"ad7316d0-07ef-4664-9345-d00cd333551d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:00.245190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.245155 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpw8\" (UniqueName: \"kubernetes.io/projected/76ef1614-c978-4046-85fe-15c48736d0da-kube-api-access-dxpw8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx\" (UID: \"76ef1614-c978-4046-85fe-15c48736d0da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:00.245370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.245253 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1aea439-9e6a-42dd-a70f-e30d47a19f19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4\" (UID: \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:00.245370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.245317 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44hf4\" (UniqueName: \"kubernetes.io/projected/ad7316d0-07ef-4664-9345-d00cd333551d-kube-api-access-44hf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m\" (UID: \"ad7316d0-07ef-4664-9345-d00cd333551d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:00.251563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.251540 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn"] Nov 02 14:30:00.255468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.255448 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:00.256512 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.256483 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr"] Nov 02 14:30:00.259226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.259207 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:00.261215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.261194 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m"] Nov 02 14:30:00.267428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.267405 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4"] Nov 02 14:30:00.270998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.270980 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:00.302217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.302193 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp"] Nov 02 14:30:00.305516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.305498 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:00.306002 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.305981 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw"] Nov 02 14:30:00.309830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.309811 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt"] Nov 02 14:30:00.309975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.309960 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:00.314361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.313948 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr"] Nov 02 14:30:00.314361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.314068 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:00.344442 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.344410 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn"] Nov 02 14:30:00.348495 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.348468 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7316d0-07ef-4664-9345-d00cd333551d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m\" (UID: \"ad7316d0-07ef-4664-9345-d00cd333551d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:00.348691 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.348672 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpw8\" (UniqueName: \"kubernetes.io/projected/76ef1614-c978-4046-85fe-15c48736d0da-kube-api-access-dxpw8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx\" (UID: \"76ef1614-c978-4046-85fe-15c48736d0da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:00.348830 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.348814 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6424d0ca-8aa6-4897-a06b-7168cd41016c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893830-h5bjw\" (UID: \"6424d0ca-8aa6-4897-a06b-7168cd41016c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:00.348956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.348941 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1aea439-9e6a-42dd-a70f-e30d47a19f19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4\" (UID: \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:00.349078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.349065 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44hf4\" (UniqueName: \"kubernetes.io/projected/ad7316d0-07ef-4664-9345-d00cd333551d-kube-api-access-44hf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m\" (UID: \"ad7316d0-07ef-4664-9345-d00cd333551d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:00.349193 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.349180 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ba8971-616e-43a8-9122-68d990259d85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt\" (UID: \"b5ba8971-616e-43a8-9122-68d990259d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:00.349314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.349300 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7gw\" (UniqueName: \"kubernetes.io/projected/b5ba8971-616e-43a8-9122-68d990259d85-kube-api-access-st7gw\") pod \"maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt\" (UID: \"b5ba8971-616e-43a8-9122-68d990259d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:00.349447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.349431 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2sx\" (UniqueName: \"kubernetes.io/projected/54b19435-f528-48b4-b807-70e9c6c80abf-kube-api-access-pt2sx\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6\" (UID: \"54b19435-f528-48b4-b807-70e9c6c80abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:00.349547 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.349531 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbg6\" (UniqueName: \"kubernetes.io/projected/3fee292c-b279-4973-9b39-2a009a781011-kube-api-access-hmbg6\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn\" (UID: \"3fee292c-b279-4973-9b39-2a009a781011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:00.349690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.349675 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p8wr\" (UniqueName: \"kubernetes.io/projected/c1aea439-9e6a-42dd-a70f-e30d47a19f19-kube-api-access-8p8wr\") pod \"maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4\" (UID: \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:00.353757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.353734 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ef1614-c978-4046-85fe-15c48736d0da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx\" (UID: \"76ef1614-c978-4046-85fe-15c48736d0da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:00.353996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.353977 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb841a6-f88a-4139-a862-78938304e503-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr\" (UID: \"fbb841a6-f88a-4139-a862-78938304e503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:00.354160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.354143 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4z7\" (UniqueName: \"kubernetes.io/projected/ce56cca9-81f2-433f-baf5-a61fb490f1f5-kube-api-access-2q4z7\") pod \"maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff\" (UID: \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:00.354309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.354291 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfdcd\" (UniqueName: \"kubernetes.io/projected/6424d0ca-8aa6-4897-a06b-7168cd41016c-kube-api-access-gfdcd\") pod \"maas-api-key-cleanup-29893830-h5bjw\" (UID: \"6424d0ca-8aa6-4897-a06b-7168cd41016c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:00.354496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.354478 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b19435-f528-48b4-b807-70e9c6c80abf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6\" (UID: \"54b19435-f528-48b4-b807-70e9c6c80abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:00.354638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.354620 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qp6l\" (UniqueName: \"kubernetes.io/projected/fbb841a6-f88a-4139-a862-78938304e503-kube-api-access-2qp6l\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr\" (UID: \"fbb841a6-f88a-4139-a862-78938304e503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:00.354769 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.354753 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fksjz\" (UniqueName: \"kubernetes.io/projected/49c1bd66-084f-49ff-87f5-4111ce798e5b-kube-api-access-fksjz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4\" (UID: \"49c1bd66-084f-49ff-87f5-4111ce798e5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:00.354978 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.354961 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce56cca9-81f2-433f-baf5-a61fb490f1f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff\" (UID: \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:00.355102 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.355089 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee0b62b-657a-4897-8e74-191f9ecb42d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp\" (UID: \"cee0b62b-657a-4897-8e74-191f9ecb42d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:00.355217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.355202 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnvv\" (UniqueName: \"kubernetes.io/projected/cee0b62b-657a-4897-8e74-191f9ecb42d6-kube-api-access-7lnvv\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp\" (UID: \"cee0b62b-657a-4897-8e74-191f9ecb42d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:00.363651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.354550 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1aea439-9e6a-42dd-a70f-e30d47a19f19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4\" (UID: \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:00.363651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.355342 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fee292c-b279-4973-9b39-2a009a781011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn\" (UID: \"3fee292c-b279-4973-9b39-2a009a781011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:00.363651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.352143 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4"] Nov 02 14:30:00.363651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.353220 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7316d0-07ef-4664-9345-d00cd333551d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m\" (UID: \"ad7316d0-07ef-4664-9345-d00cd333551d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:00.363651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.357345 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp"] Nov 02 14:30:00.378971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.370996 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b19435-f528-48b4-b807-70e9c6c80abf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6\" (UID: \"54b19435-f528-48b4-b807-70e9c6c80abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:00.378971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.371698 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce56cca9-81f2-433f-baf5-a61fb490f1f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff\" (UID: \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:00.378971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.371804 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c1bd66-084f-49ff-87f5-4111ce798e5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4\" (UID: \"49c1bd66-084f-49ff-87f5-4111ce798e5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:00.378971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.373126 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ef1614-c978-4046-85fe-15c48736d0da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx\" (UID: \"76ef1614-c978-4046-85fe-15c48736d0da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:00.384902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.384874 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44hf4\" (UniqueName: \"kubernetes.io/projected/ad7316d0-07ef-4664-9345-d00cd333551d-kube-api-access-44hf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m\" (UID: \"ad7316d0-07ef-4664-9345-d00cd333551d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:00.387654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.387630 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpw8\" (UniqueName: \"kubernetes.io/projected/76ef1614-c978-4046-85fe-15c48736d0da-kube-api-access-dxpw8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx\" (UID: \"76ef1614-c978-4046-85fe-15c48736d0da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:00.389259 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.389231 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw"] Nov 02 14:30:00.393161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.393135 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt"] Nov 02 14:30:00.409968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.409943 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4z7\" (UniqueName: \"kubernetes.io/projected/ce56cca9-81f2-433f-baf5-a61fb490f1f5-kube-api-access-2q4z7\") pod \"maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff\" (UID: \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:00.412335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.412271 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2sx\" (UniqueName: \"kubernetes.io/projected/54b19435-f528-48b4-b807-70e9c6c80abf-kube-api-access-pt2sx\") pod \"maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6\" (UID: \"54b19435-f528-48b4-b807-70e9c6c80abf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:00.416298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.416280 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p8wr\" (UniqueName: \"kubernetes.io/projected/c1aea439-9e6a-42dd-a70f-e30d47a19f19-kube-api-access-8p8wr\") pod \"maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4\" (UID: \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:00.475058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475027 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb841a6-f88a-4139-a862-78938304e503-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr\" (UID: \"fbb841a6-f88a-4139-a862-78938304e503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:00.475194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475078 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdcd\" (UniqueName: \"kubernetes.io/projected/6424d0ca-8aa6-4897-a06b-7168cd41016c-kube-api-access-gfdcd\") pod \"maas-api-key-cleanup-29893830-h5bjw\" (UID: \"6424d0ca-8aa6-4897-a06b-7168cd41016c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:00.475194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475140 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qp6l\" (UniqueName: \"kubernetes.io/projected/fbb841a6-f88a-4139-a862-78938304e503-kube-api-access-2qp6l\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr\" (UID: \"fbb841a6-f88a-4139-a862-78938304e503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:00.475194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475169 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fksjz\" (UniqueName: \"kubernetes.io/projected/49c1bd66-084f-49ff-87f5-4111ce798e5b-kube-api-access-fksjz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4\" (UID: \"49c1bd66-084f-49ff-87f5-4111ce798e5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:00.475361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475306 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee0b62b-657a-4897-8e74-191f9ecb42d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp\" (UID: \"cee0b62b-657a-4897-8e74-191f9ecb42d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:00.475361 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475344 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnvv\" (UniqueName: \"kubernetes.io/projected/cee0b62b-657a-4897-8e74-191f9ecb42d6-kube-api-access-7lnvv\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp\" (UID: \"cee0b62b-657a-4897-8e74-191f9ecb42d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:00.475479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475376 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fee292c-b279-4973-9b39-2a009a781011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn\" (UID: \"3fee292c-b279-4973-9b39-2a009a781011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:00.475479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475397 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c1bd66-084f-49ff-87f5-4111ce798e5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4\" (UID: \"49c1bd66-084f-49ff-87f5-4111ce798e5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:00.475479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475436 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6424d0ca-8aa6-4897-a06b-7168cd41016c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893830-h5bjw\" (UID: \"6424d0ca-8aa6-4897-a06b-7168cd41016c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:00.475645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475490 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ba8971-616e-43a8-9122-68d990259d85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt\" (UID: \"b5ba8971-616e-43a8-9122-68d990259d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:00.475645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475533 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7gw\" (UniqueName: \"kubernetes.io/projected/b5ba8971-616e-43a8-9122-68d990259d85-kube-api-access-st7gw\") pod \"maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt\" (UID: \"b5ba8971-616e-43a8-9122-68d990259d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:00.475645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475590 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbg6\" (UniqueName: \"kubernetes.io/projected/3fee292c-b279-4973-9b39-2a009a781011-kube-api-access-hmbg6\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn\" (UID: \"3fee292c-b279-4973-9b39-2a009a781011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:00.475962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.475932 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb841a6-f88a-4139-a862-78938304e503-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr\" (UID: \"fbb841a6-f88a-4139-a862-78938304e503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:00.476266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.476245 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fee292c-b279-4973-9b39-2a009a781011-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn\" (UID: \"3fee292c-b279-4973-9b39-2a009a781011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:00.476370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.476340 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ba8971-616e-43a8-9122-68d990259d85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt\" (UID: \"b5ba8971-616e-43a8-9122-68d990259d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:00.476370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.476342 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee0b62b-657a-4897-8e74-191f9ecb42d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp\" (UID: \"cee0b62b-657a-4897-8e74-191f9ecb42d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:00.476483 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.476341 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c1bd66-084f-49ff-87f5-4111ce798e5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4\" (UID: \"49c1bd66-084f-49ff-87f5-4111ce798e5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:00.476483 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.476427 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6424d0ca-8aa6-4897-a06b-7168cd41016c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893830-h5bjw\" (UID: \"6424d0ca-8aa6-4897-a06b-7168cd41016c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:00.504870 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.504845 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fksjz\" (UniqueName: \"kubernetes.io/projected/49c1bd66-084f-49ff-87f5-4111ce798e5b-kube-api-access-fksjz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4\" (UID: \"49c1bd66-084f-49ff-87f5-4111ce798e5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:00.507549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.507531 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:00.510919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.510895 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7gw\" (UniqueName: \"kubernetes.io/projected/b5ba8971-616e-43a8-9122-68d990259d85-kube-api-access-st7gw\") pod \"maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt\" (UID: \"b5ba8971-616e-43a8-9122-68d990259d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:00.511015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.510977 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qp6l\" (UniqueName: \"kubernetes.io/projected/fbb841a6-f88a-4139-a862-78938304e503-kube-api-access-2qp6l\") pod \"maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr\" (UID: \"fbb841a6-f88a-4139-a862-78938304e503\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:00.515047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.515023 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfdcd\" (UniqueName: \"kubernetes.io/projected/6424d0ca-8aa6-4897-a06b-7168cd41016c-kube-api-access-gfdcd\") pod \"maas-api-key-cleanup-29893830-h5bjw\" (UID: \"6424d0ca-8aa6-4897-a06b-7168cd41016c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:00.516161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.516142 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnvv\" (UniqueName: \"kubernetes.io/projected/cee0b62b-657a-4897-8e74-191f9ecb42d6-kube-api-access-7lnvv\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp\" (UID: \"cee0b62b-657a-4897-8e74-191f9ecb42d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:00.516732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.516714 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbg6\" (UniqueName: \"kubernetes.io/projected/3fee292c-b279-4973-9b39-2a009a781011-kube-api-access-hmbg6\") pod \"maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn\" (UID: \"3fee292c-b279-4973-9b39-2a009a781011\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:00.521751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.521733 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:00.528609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.528591 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:30:00.535298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.535278 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:00.553368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.553170 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:00.563963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.563366 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:00.573735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.573552 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:00.596984 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.581922 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:00.596984 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.592280 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:00.622914 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.622823 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:00.635662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.635606 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:30:00.672231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.650730 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:00.713500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.693491 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:00.836588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:00.836555 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6"] Nov 02 14:30:00.839428 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:00.839401 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b19435_f528_48b4_b807_70e9c6c80abf.slice/crio-40fb456f901da1a1b1f65973c89ed72d8c69b01221cb54365a57b40cce36d6f8 WatchSource:0}: Error finding container 40fb456f901da1a1b1f65973c89ed72d8c69b01221cb54365a57b40cce36d6f8: Status 404 returned error can't find the container with id 40fb456f901da1a1b1f65973c89ed72d8c69b01221cb54365a57b40cce36d6f8 Nov 02 14:30:01.047899 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.047873 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx"] Nov 02 14:30:01.048155 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:01.048133 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ef1614_c978_4046_85fe_15c48736d0da.slice/crio-140e6823b146774cc366da5cedb6b0474a14e112ef90060b0a7a675d5d83d435 WatchSource:0}: Error finding container 140e6823b146774cc366da5cedb6b0474a14e112ef90060b0a7a675d5d83d435: Status 404 returned error can't find the container with id 140e6823b146774cc366da5cedb6b0474a14e112ef90060b0a7a675d5d83d435 Nov 02 14:30:01.051162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.051135 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" event={"ID":"54b19435-f528-48b4-b807-70e9c6c80abf","Type":"ContainerStarted","Data":"40fb456f901da1a1b1f65973c89ed72d8c69b01221cb54365a57b40cce36d6f8"} Nov 02 14:30:01.052859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.052836 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:30:01.084656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.084634 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4"] Nov 02 14:30:01.088190 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:01.088165 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1aea439_9e6a_42dd_a70f_e30d47a19f19.slice/crio-ac793e8dc184c7d7c782a05f72ab440890ca97ab2d31aa7537608d204a5fba98 WatchSource:0}: Error finding container ac793e8dc184c7d7c782a05f72ab440890ca97ab2d31aa7537608d204a5fba98: Status 404 returned error can't find the container with id ac793e8dc184c7d7c782a05f72ab440890ca97ab2d31aa7537608d204a5fba98 Nov 02 14:30:01.089672 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.089647 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt"] Nov 02 14:30:01.118042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.118015 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw"] Nov 02 14:30:01.154806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.154751 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn"] Nov 02 14:30:01.160140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.160119 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m"] Nov 02 14:30:01.178425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.176370 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff"] Nov 02 14:30:01.192636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.192613 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp"] Nov 02 14:30:01.206913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.206891 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4"] Nov 02 14:30:01.228902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:01.228880 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr"] Nov 02 14:30:01.274929 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:01.274887 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce56cca9_81f2_433f_baf5_a61fb490f1f5.slice/crio-4f89a35cff06e92850a3369300c19702dc8015c23d3ed3bde30ee44880ca2b80 WatchSource:0}: Error finding container 4f89a35cff06e92850a3369300c19702dc8015c23d3ed3bde30ee44880ca2b80: Status 404 returned error can't find the container with id 4f89a35cff06e92850a3369300c19702dc8015c23d3ed3bde30ee44880ca2b80 Nov 02 14:30:02.141188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.141138 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" event={"ID":"3fee292c-b279-4973-9b39-2a009a781011","Type":"ContainerStarted","Data":"5adb1fc6effb6136f7387cad7b961292b85b1ac5525f3770a596d03da16ee53a"} Nov 02 14:30:02.141692 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.141209 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" event={"ID":"3fee292c-b279-4973-9b39-2a009a781011","Type":"ContainerStarted","Data":"5dc771c01656fe9106654559cff4837c048da8f9ebe352b8f7e0da82e9eb3faf"} Nov 02 14:30:02.144077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.144043 4431 generic.go:356] "Generic (PLEG): container finished" podID="54b19435-f528-48b4-b807-70e9c6c80abf" containerID="e54a5b7ec116a4ab7654958fe1df915a8fc7d45b985e9283348f27ff7278df33" exitCode=60 Nov 02 14:30:02.144424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.144179 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" event={"ID":"54b19435-f528-48b4-b807-70e9c6c80abf","Type":"ContainerDied","Data":"e54a5b7ec116a4ab7654958fe1df915a8fc7d45b985e9283348f27ff7278df33"} Nov 02 14:30:02.144507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.144454 4431 scope.go:122] "RemoveContainer" containerID="e54a5b7ec116a4ab7654958fe1df915a8fc7d45b985e9283348f27ff7278df33" Nov 02 14:30:02.147734 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.147695 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" event={"ID":"b5ba8971-616e-43a8-9122-68d990259d85","Type":"ContainerStarted","Data":"cb77a0cb364b2cbdd6985b4a1ab3c4c9f2c49358e343fc1c57cd89b514f0605f"} Nov 02 14:30:02.147924 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.147906 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" event={"ID":"b5ba8971-616e-43a8-9122-68d990259d85","Type":"ContainerStarted","Data":"77afa060bebc945b8683a79017498684fb0134bc7c6b121e8357d2cf09ad2420"} Nov 02 14:30:02.151465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.151441 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" event={"ID":"ce56cca9-81f2-433f-baf5-a61fb490f1f5","Type":"ContainerStarted","Data":"78307d98e68dfcd2b3aa5a2d47a9132bc9010582cfec7c7943144756e2c6a05d"} Nov 02 14:30:02.151642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.151623 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" event={"ID":"ce56cca9-81f2-433f-baf5-a61fb490f1f5","Type":"ContainerStarted","Data":"4f89a35cff06e92850a3369300c19702dc8015c23d3ed3bde30ee44880ca2b80"} Nov 02 14:30:02.155114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.155094 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" event={"ID":"ad7316d0-07ef-4664-9345-d00cd333551d","Type":"ContainerStarted","Data":"01895f8fa21bdd272cc3d4e820007e1df0bd13098daa372394d643ca3e7b4c3b"} Nov 02 14:30:02.155288 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.155259 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" event={"ID":"ad7316d0-07ef-4664-9345-d00cd333551d","Type":"ContainerStarted","Data":"0016a68d2861289f8cc268a32986d6d2bb502586f2bd97ece2901d202b4d0e3a"} Nov 02 14:30:02.158007 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.157983 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" event={"ID":"cee0b62b-657a-4897-8e74-191f9ecb42d6","Type":"ContainerStarted","Data":"3d96aed95dafa4273488b7aba53c34bfbcf75ed962722e09bc586ffc568717f8"} Nov 02 14:30:02.159057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.159037 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" event={"ID":"cee0b62b-657a-4897-8e74-191f9ecb42d6","Type":"ContainerStarted","Data":"8939fbe0db84127ea9a4a986513afc7ada7ceaf970c9b43190cd19eb4db63036"} Nov 02 14:30:02.161268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.161248 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" event={"ID":"49c1bd66-084f-49ff-87f5-4111ce798e5b","Type":"ContainerStarted","Data":"c6619a4b76aeb1fd1a7629eb45978fbfd1e30ef1a1b6457845d945e6caabc39e"} Nov 02 14:30:02.161450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.161436 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" event={"ID":"49c1bd66-084f-49ff-87f5-4111ce798e5b","Type":"ContainerStarted","Data":"b39df9383885bdd79e037426a664b6b9335c99cf8dc4180111ce41c54864db56"} Nov 02 14:30:02.163152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.163109 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" event={"ID":"c1aea439-9e6a-42dd-a70f-e30d47a19f19","Type":"ContainerStarted","Data":"a9a9c9ed9d336e6975997c86f13371682e9f732541019306d380bc97c6a4e0c0"} Nov 02 14:30:02.164163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.164145 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" event={"ID":"c1aea439-9e6a-42dd-a70f-e30d47a19f19","Type":"ContainerStarted","Data":"ac793e8dc184c7d7c782a05f72ab440890ca97ab2d31aa7537608d204a5fba98"} Nov 02 14:30:02.165810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.165271 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" event={"ID":"76ef1614-c978-4046-85fe-15c48736d0da","Type":"ContainerStarted","Data":"547a42382980965aebf5fea0fc4b7fb23cee2c38ee8364e32d78c95bec6f7f52"} Nov 02 14:30:02.165810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.165320 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" event={"ID":"76ef1614-c978-4046-85fe-15c48736d0da","Type":"ContainerStarted","Data":"140e6823b146774cc366da5cedb6b0474a14e112ef90060b0a7a675d5d83d435"} Nov 02 14:30:02.168537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.168513 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" event={"ID":"fbb841a6-f88a-4139-a862-78938304e503","Type":"ContainerStarted","Data":"5f45d78227f48e14e97c41fe9cfd2fa3c096fa7b90a6de515ac8844b8ee52f41"} Nov 02 14:30:02.168613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.168556 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" event={"ID":"fbb841a6-f88a-4139-a862-78938304e503","Type":"ContainerStarted","Data":"b189d23b4332f8b5b5f7691b568288f60327234637e70a359f8ca7e8a208e4dc"} Nov 02 14:30:02.189137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.189106 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" event={"ID":"6424d0ca-8aa6-4897-a06b-7168cd41016c","Type":"ContainerStarted","Data":"39676b92cca8181ab783a5a672c341383c6f8e01d56685a3aad4d68f3fe217b8"} Nov 02 14:30:02.189258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.189162 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" event={"ID":"6424d0ca-8aa6-4897-a06b-7168cd41016c","Type":"ContainerStarted","Data":"d5f29d0fc2fd7f9ce2035c88f8f97879cdd45c13268bdd67092fc891e5e8f571"} Nov 02 14:30:02.189258 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.189178 4431 scope.go:122] "RemoveContainer" containerID="39676b92cca8181ab783a5a672c341383c6f8e01d56685a3aad4d68f3fe217b8" Nov 02 14:30:02.190671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.190623 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" podStartSLOduration=2.190607385 podStartE2EDuration="2.190607385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.186763801 +0000 UTC m=+1967.223868735" watchObservedRunningTime="2026-11-02 14:30:02.190607385 +0000 UTC m=+1967.227712319" Nov 02 14:30:02.229183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.229139 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" podStartSLOduration=2.229125021 podStartE2EDuration="2.229125021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.228469442 +0000 UTC m=+1967.265574371" watchObservedRunningTime="2026-11-02 14:30:02.229125021 +0000 UTC m=+1967.266229947" Nov 02 14:30:02.271324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.271262 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" podStartSLOduration=2.271245783 podStartE2EDuration="2.271245783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.27082742 +0000 UTC m=+1967.307932354" watchObservedRunningTime="2026-11-02 14:30:02.271245783 +0000 UTC m=+1967.308350716" Nov 02 14:30:02.315133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.315079 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" podStartSLOduration=2.3150663 podStartE2EDuration="2.3150663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.313177774 +0000 UTC m=+1967.350282706" watchObservedRunningTime="2026-11-02 14:30:02.3150663 +0000 UTC m=+1967.352171254" Nov 02 14:30:02.358177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.358127 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" podStartSLOduration=2.358113141 podStartE2EDuration="2.358113141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.355683096 +0000 UTC m=+1967.392788028" watchObservedRunningTime="2026-11-02 14:30:02.358113141 +0000 UTC m=+1967.395218073" Nov 02 14:30:02.404062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.403959 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" podStartSLOduration=2.403943767 podStartE2EDuration="2.403943767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.399913658 +0000 UTC m=+1967.437018591" watchObservedRunningTime="2026-11-02 14:30:02.403943767 +0000 UTC m=+1967.441048715" Nov 02 14:30:02.447613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.447558 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" podStartSLOduration=2.447543886 podStartE2EDuration="2.447543886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.444448273 +0000 UTC m=+1967.481553206" watchObservedRunningTime="2026-11-02 14:30:02.447543886 +0000 UTC m=+1967.484648863" Nov 02 14:30:02.531924 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.531868 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" podStartSLOduration=2.531854191 podStartE2EDuration="2.531854191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.528184593 +0000 UTC m=+1967.565289538" watchObservedRunningTime="2026-11-02 14:30:02.531854191 +0000 UTC m=+1967.568959123" Nov 02 14:30:02.574541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:02.574492 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" podStartSLOduration=2.574478885 podStartE2EDuration="2.574478885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:02.569579676 +0000 UTC m=+1967.606684612" watchObservedRunningTime="2026-11-02 14:30:02.574478885 +0000 UTC m=+1967.611583817" Nov 02 14:30:03.205573 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.205533 4431 generic.go:356] "Generic (PLEG): container finished" podID="3fee292c-b279-4973-9b39-2a009a781011" containerID="5adb1fc6effb6136f7387cad7b961292b85b1ac5525f3770a596d03da16ee53a" exitCode=60 Nov 02 14:30:03.206221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.206189 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" event={"ID":"3fee292c-b279-4973-9b39-2a009a781011","Type":"ContainerDied","Data":"5adb1fc6effb6136f7387cad7b961292b85b1ac5525f3770a596d03da16ee53a"} Nov 02 14:30:03.206401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.206381 4431 scope.go:122] "RemoveContainer" containerID="5adb1fc6effb6136f7387cad7b961292b85b1ac5525f3770a596d03da16ee53a" Nov 02 14:30:03.208817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.208759 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" event={"ID":"54b19435-f528-48b4-b807-70e9c6c80abf","Type":"ContainerStarted","Data":"c042458cbfe36b61fee29b113c559c3affea12611f33dc9e59f35a3188daee82"} Nov 02 14:30:03.210610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.210583 4431 generic.go:356] "Generic (PLEG): container finished" podID="b5ba8971-616e-43a8-9122-68d990259d85" containerID="cb77a0cb364b2cbdd6985b4a1ab3c4c9f2c49358e343fc1c57cd89b514f0605f" exitCode=60 Nov 02 14:30:03.210718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.210664 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" event={"ID":"b5ba8971-616e-43a8-9122-68d990259d85","Type":"ContainerDied","Data":"cb77a0cb364b2cbdd6985b4a1ab3c4c9f2c49358e343fc1c57cd89b514f0605f"} Nov 02 14:30:03.211143 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.211122 4431 scope.go:122] "RemoveContainer" containerID="cb77a0cb364b2cbdd6985b4a1ab3c4c9f2c49358e343fc1c57cd89b514f0605f" Nov 02 14:30:03.212586 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.212565 4431 generic.go:356] "Generic (PLEG): container finished" podID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerID="78307d98e68dfcd2b3aa5a2d47a9132bc9010582cfec7c7943144756e2c6a05d" exitCode=60 Nov 02 14:30:03.212698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.212614 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" event={"ID":"ce56cca9-81f2-433f-baf5-a61fb490f1f5","Type":"ContainerDied","Data":"78307d98e68dfcd2b3aa5a2d47a9132bc9010582cfec7c7943144756e2c6a05d"} Nov 02 14:30:03.212903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.212886 4431 scope.go:122] "RemoveContainer" containerID="78307d98e68dfcd2b3aa5a2d47a9132bc9010582cfec7c7943144756e2c6a05d" Nov 02 14:30:03.214375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.214355 4431 generic.go:356] "Generic (PLEG): container finished" podID="ad7316d0-07ef-4664-9345-d00cd333551d" containerID="01895f8fa21bdd272cc3d4e820007e1df0bd13098daa372394d643ca3e7b4c3b" exitCode=60 Nov 02 14:30:03.214461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.214381 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" event={"ID":"ad7316d0-07ef-4664-9345-d00cd333551d","Type":"ContainerDied","Data":"01895f8fa21bdd272cc3d4e820007e1df0bd13098daa372394d643ca3e7b4c3b"} Nov 02 14:30:03.214633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.214618 4431 scope.go:122] "RemoveContainer" containerID="01895f8fa21bdd272cc3d4e820007e1df0bd13098daa372394d643ca3e7b4c3b" Nov 02 14:30:03.216221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.216202 4431 generic.go:356] "Generic (PLEG): container finished" podID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerID="3d96aed95dafa4273488b7aba53c34bfbcf75ed962722e09bc586ffc568717f8" exitCode=60 Nov 02 14:30:03.216327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.216284 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" event={"ID":"cee0b62b-657a-4897-8e74-191f9ecb42d6","Type":"ContainerDied","Data":"3d96aed95dafa4273488b7aba53c34bfbcf75ed962722e09bc586ffc568717f8"} Nov 02 14:30:03.216469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.216452 4431 scope.go:122] "RemoveContainer" containerID="3d96aed95dafa4273488b7aba53c34bfbcf75ed962722e09bc586ffc568717f8" Nov 02 14:30:03.218474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.218450 4431 generic.go:356] "Generic (PLEG): container finished" podID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerID="c6619a4b76aeb1fd1a7629eb45978fbfd1e30ef1a1b6457845d945e6caabc39e" exitCode=60 Nov 02 14:30:03.218584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.218479 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" event={"ID":"49c1bd66-084f-49ff-87f5-4111ce798e5b","Type":"ContainerDied","Data":"c6619a4b76aeb1fd1a7629eb45978fbfd1e30ef1a1b6457845d945e6caabc39e"} Nov 02 14:30:03.218712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.218695 4431 scope.go:122] "RemoveContainer" containerID="c6619a4b76aeb1fd1a7629eb45978fbfd1e30ef1a1b6457845d945e6caabc39e" Nov 02 14:30:03.221621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.221594 4431 generic.go:356] "Generic (PLEG): container finished" podID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerID="a9a9c9ed9d336e6975997c86f13371682e9f732541019306d380bc97c6a4e0c0" exitCode=60 Nov 02 14:30:03.221715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.221668 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" event={"ID":"c1aea439-9e6a-42dd-a70f-e30d47a19f19","Type":"ContainerDied","Data":"a9a9c9ed9d336e6975997c86f13371682e9f732541019306d380bc97c6a4e0c0"} Nov 02 14:30:03.223260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.223243 4431 scope.go:122] "RemoveContainer" containerID="a9a9c9ed9d336e6975997c86f13371682e9f732541019306d380bc97c6a4e0c0" Nov 02 14:30:03.225200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.225179 4431 generic.go:356] "Generic (PLEG): container finished" podID="76ef1614-c978-4046-85fe-15c48736d0da" containerID="547a42382980965aebf5fea0fc4b7fb23cee2c38ee8364e32d78c95bec6f7f52" exitCode=60 Nov 02 14:30:03.225280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.225265 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" event={"ID":"76ef1614-c978-4046-85fe-15c48736d0da","Type":"ContainerDied","Data":"547a42382980965aebf5fea0fc4b7fb23cee2c38ee8364e32d78c95bec6f7f52"} Nov 02 14:30:03.225579 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.225535 4431 scope.go:122] "RemoveContainer" containerID="547a42382980965aebf5fea0fc4b7fb23cee2c38ee8364e32d78c95bec6f7f52" Nov 02 14:30:03.235731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.235707 4431 generic.go:356] "Generic (PLEG): container finished" podID="fbb841a6-f88a-4139-a862-78938304e503" containerID="5f45d78227f48e14e97c41fe9cfd2fa3c096fa7b90a6de515ac8844b8ee52f41" exitCode=60 Nov 02 14:30:03.235964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.235946 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" event={"ID":"fbb841a6-f88a-4139-a862-78938304e503","Type":"ContainerDied","Data":"5f45d78227f48e14e97c41fe9cfd2fa3c096fa7b90a6de515ac8844b8ee52f41"} Nov 02 14:30:03.236589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.236570 4431 scope.go:122] "RemoveContainer" containerID="5f45d78227f48e14e97c41fe9cfd2fa3c096fa7b90a6de515ac8844b8ee52f41" Nov 02 14:30:03.242173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.242138 4431 generic.go:356] "Generic (PLEG): container finished" podID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerID="39676b92cca8181ab783a5a672c341383c6f8e01d56685a3aad4d68f3fe217b8" exitCode=60 Nov 02 14:30:03.242310 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.242291 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" event={"ID":"6424d0ca-8aa6-4897-a06b-7168cd41016c","Type":"ContainerDied","Data":"39676b92cca8181ab783a5a672c341383c6f8e01d56685a3aad4d68f3fe217b8"} Nov 02 14:30:03.242384 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.242340 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" event={"ID":"6424d0ca-8aa6-4897-a06b-7168cd41016c","Type":"ContainerStarted","Data":"4c7c17df18673edfaea4bb8917c27debf6297cae18ed07edd863e049fa8d6263"} Nov 02 14:30:03.395416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.395365 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" podStartSLOduration=3.395350804 podStartE2EDuration="3.395350804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:03.392400014 +0000 UTC m=+1968.429504949" watchObservedRunningTime="2026-11-02 14:30:03.395350804 +0000 UTC m=+1968.432455790" Nov 02 14:30:03.705297 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:03.705240 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" podStartSLOduration=3.705222299 podStartE2EDuration="3.705222299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:03.700655984 +0000 UTC m=+1968.737760916" watchObservedRunningTime="2026-11-02 14:30:03.705222299 +0000 UTC m=+1968.742327231" Nov 02 14:30:04.250621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.250574 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" event={"ID":"fbb841a6-f88a-4139-a862-78938304e503","Type":"ContainerStarted","Data":"53d633eea425be2a3a38f25b2f13b9463a0be00c6fbe49e6a6b74e8be7e6bdc6"} Nov 02 14:30:04.252631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.252603 4431 generic.go:356] "Generic (PLEG): container finished" podID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerID="4c7c17df18673edfaea4bb8917c27debf6297cae18ed07edd863e049fa8d6263" exitCode=60 Nov 02 14:30:04.252820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.252685 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" event={"ID":"6424d0ca-8aa6-4897-a06b-7168cd41016c","Type":"ContainerDied","Data":"4c7c17df18673edfaea4bb8917c27debf6297cae18ed07edd863e049fa8d6263"} Nov 02 14:30:04.252820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.252742 4431 scope.go:122] "RemoveContainer" containerID="39676b92cca8181ab783a5a672c341383c6f8e01d56685a3aad4d68f3fe217b8" Nov 02 14:30:04.252947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.252894 4431 scope.go:122] "RemoveContainer" containerID="4c7c17df18673edfaea4bb8917c27debf6297cae18ed07edd863e049fa8d6263" Nov 02 14:30:04.253153 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:04.253121 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893830-h5bjw_odh-ai-gateway-infra(6424d0ca-8aa6-4897-a06b-7168cd41016c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" Nov 02 14:30:04.255055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.255034 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" event={"ID":"3fee292c-b279-4973-9b39-2a009a781011","Type":"ContainerStarted","Data":"406bb317623af6579305afdc14e73c27d1e92f810bad8644f5ba2c6f28b8502e"} Nov 02 14:30:04.257005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.256985 4431 generic.go:356] "Generic (PLEG): container finished" podID="54b19435-f528-48b4-b807-70e9c6c80abf" containerID="c042458cbfe36b61fee29b113c559c3affea12611f33dc9e59f35a3188daee82" exitCode=60 Nov 02 14:30:04.257113 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.257056 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" event={"ID":"54b19435-f528-48b4-b807-70e9c6c80abf","Type":"ContainerDied","Data":"c042458cbfe36b61fee29b113c559c3affea12611f33dc9e59f35a3188daee82"} Nov 02 14:30:04.257262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.257246 4431 scope.go:122] "RemoveContainer" containerID="c042458cbfe36b61fee29b113c559c3affea12611f33dc9e59f35a3188daee82" Nov 02 14:30:04.257499 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:04.257481 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6_odh-ai-gateway-infra(54b19435-f528-48b4-b807-70e9c6c80abf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" Nov 02 14:30:04.258994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.258966 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" event={"ID":"b5ba8971-616e-43a8-9122-68d990259d85","Type":"ContainerStarted","Data":"6266b6cef9ddd66ab6cd029aab2091c515c9d4a679cbe3e86602a340f32db1f4"} Nov 02 14:30:04.261058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.261028 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" event={"ID":"ce56cca9-81f2-433f-baf5-a61fb490f1f5","Type":"ContainerStarted","Data":"ed70e2f2e532d2dd1e4d6895a85f7e97b6f2abf5105b922289cb74b590869a0d"} Nov 02 14:30:04.263810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.263790 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" event={"ID":"ad7316d0-07ef-4664-9345-d00cd333551d","Type":"ContainerStarted","Data":"037765bb70775157d8338ea128d2fada689dbbc83fdcb9d78ed2c6328d29c41e"} Nov 02 14:30:04.265856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.265831 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" event={"ID":"cee0b62b-657a-4897-8e74-191f9ecb42d6","Type":"ContainerStarted","Data":"bbd3e5b23336a04451c99ac167038a4e3e6d2267c4c2ada29256fc083c323035"} Nov 02 14:30:04.266633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.266616 4431 scope.go:122] "RemoveContainer" containerID="e54a5b7ec116a4ab7654958fe1df915a8fc7d45b985e9283348f27ff7278df33" Nov 02 14:30:04.268032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.268008 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" event={"ID":"49c1bd66-084f-49ff-87f5-4111ce798e5b","Type":"ContainerStarted","Data":"739b5e2168c0644b159fb3efb2a562b6b175a165e7e13ddc73ca9b241c927c21"} Nov 02 14:30:04.269842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.269815 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" event={"ID":"c1aea439-9e6a-42dd-a70f-e30d47a19f19","Type":"ContainerStarted","Data":"9fceb5fb01643afcf2d7ee44f331d0bac762fe3080fe433ef374226d25726e61"} Nov 02 14:30:04.272372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:04.272350 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" event={"ID":"76ef1614-c978-4046-85fe-15c48736d0da","Type":"ContainerStarted","Data":"02e22d58f2eedc261f725019bb49a22b9cbf15cb1afdc22c592b843b612b3d87"} Nov 02 14:30:05.282280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.282241 4431 generic.go:356] "Generic (PLEG): container finished" podID="3fee292c-b279-4973-9b39-2a009a781011" containerID="406bb317623af6579305afdc14e73c27d1e92f810bad8644f5ba2c6f28b8502e" exitCode=60 Nov 02 14:30:05.282754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.282319 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" event={"ID":"3fee292c-b279-4973-9b39-2a009a781011","Type":"ContainerDied","Data":"406bb317623af6579305afdc14e73c27d1e92f810bad8644f5ba2c6f28b8502e"} Nov 02 14:30:05.282754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.282385 4431 scope.go:122] "RemoveContainer" containerID="5adb1fc6effb6136f7387cad7b961292b85b1ac5525f3770a596d03da16ee53a" Nov 02 14:30:05.282754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.282688 4431 scope.go:122] "RemoveContainer" containerID="406bb317623af6579305afdc14e73c27d1e92f810bad8644f5ba2c6f28b8502e" Nov 02 14:30:05.283015 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.282994 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn_odh-ai-gateway-infra(3fee292c-b279-4973-9b39-2a009a781011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" podUID="3fee292c-b279-4973-9b39-2a009a781011" Nov 02 14:30:05.285019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.284997 4431 scope.go:122] "RemoveContainer" containerID="c042458cbfe36b61fee29b113c559c3affea12611f33dc9e59f35a3188daee82" Nov 02 14:30:05.285317 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.285296 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6_odh-ai-gateway-infra(54b19435-f528-48b4-b807-70e9c6c80abf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" Nov 02 14:30:05.286704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.286685 4431 generic.go:356] "Generic (PLEG): container finished" podID="b5ba8971-616e-43a8-9122-68d990259d85" containerID="6266b6cef9ddd66ab6cd029aab2091c515c9d4a679cbe3e86602a340f32db1f4" exitCode=60 Nov 02 14:30:05.286834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.286738 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" event={"ID":"b5ba8971-616e-43a8-9122-68d990259d85","Type":"ContainerDied","Data":"6266b6cef9ddd66ab6cd029aab2091c515c9d4a679cbe3e86602a340f32db1f4"} Nov 02 14:30:05.287087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.287070 4431 scope.go:122] "RemoveContainer" containerID="6266b6cef9ddd66ab6cd029aab2091c515c9d4a679cbe3e86602a340f32db1f4" Nov 02 14:30:05.287393 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.287367 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt_odh-ai-gateway-infra(b5ba8971-616e-43a8-9122-68d990259d85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" podUID="b5ba8971-616e-43a8-9122-68d990259d85" Nov 02 14:30:05.288748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.288726 4431 generic.go:356] "Generic (PLEG): container finished" podID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerID="ed70e2f2e532d2dd1e4d6895a85f7e97b6f2abf5105b922289cb74b590869a0d" exitCode=60 Nov 02 14:30:05.288864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.288817 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" event={"ID":"ce56cca9-81f2-433f-baf5-a61fb490f1f5","Type":"ContainerDied","Data":"ed70e2f2e532d2dd1e4d6895a85f7e97b6f2abf5105b922289cb74b590869a0d"} Nov 02 14:30:05.289052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.289036 4431 scope.go:122] "RemoveContainer" containerID="ed70e2f2e532d2dd1e4d6895a85f7e97b6f2abf5105b922289cb74b590869a0d" Nov 02 14:30:05.289335 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.289312 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff_odh-ai-gateway-infra(ce56cca9-81f2-433f-baf5-a61fb490f1f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" Nov 02 14:30:05.291056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.291037 4431 generic.go:356] "Generic (PLEG): container finished" podID="ad7316d0-07ef-4664-9345-d00cd333551d" containerID="037765bb70775157d8338ea128d2fada689dbbc83fdcb9d78ed2c6328d29c41e" exitCode=60 Nov 02 14:30:05.291154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.291060 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" event={"ID":"ad7316d0-07ef-4664-9345-d00cd333551d","Type":"ContainerDied","Data":"037765bb70775157d8338ea128d2fada689dbbc83fdcb9d78ed2c6328d29c41e"} Nov 02 14:30:05.291339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.291322 4431 scope.go:122] "RemoveContainer" containerID="037765bb70775157d8338ea128d2fada689dbbc83fdcb9d78ed2c6328d29c41e" Nov 02 14:30:05.291622 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.291596 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m_odh-ai-gateway-infra(ad7316d0-07ef-4664-9345-d00cd333551d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" Nov 02 14:30:05.293053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.293034 4431 generic.go:356] "Generic (PLEG): container finished" podID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerID="bbd3e5b23336a04451c99ac167038a4e3e6d2267c4c2ada29256fc083c323035" exitCode=60 Nov 02 14:30:05.293150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.293125 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" event={"ID":"cee0b62b-657a-4897-8e74-191f9ecb42d6","Type":"ContainerDied","Data":"bbd3e5b23336a04451c99ac167038a4e3e6d2267c4c2ada29256fc083c323035"} Nov 02 14:30:05.293291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.293276 4431 scope.go:122] "RemoveContainer" containerID="bbd3e5b23336a04451c99ac167038a4e3e6d2267c4c2ada29256fc083c323035" Nov 02 14:30:05.293491 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.293471 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp_odh-ai-gateway-infra(cee0b62b-657a-4897-8e74-191f9ecb42d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" Nov 02 14:30:05.294957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.294938 4431 generic.go:356] "Generic (PLEG): container finished" podID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerID="739b5e2168c0644b159fb3efb2a562b6b175a165e7e13ddc73ca9b241c927c21" exitCode=60 Nov 02 14:30:05.295037 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.294993 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" event={"ID":"49c1bd66-084f-49ff-87f5-4111ce798e5b","Type":"ContainerDied","Data":"739b5e2168c0644b159fb3efb2a562b6b175a165e7e13ddc73ca9b241c927c21"} Nov 02 14:30:05.295186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.295173 4431 scope.go:122] "RemoveContainer" containerID="739b5e2168c0644b159fb3efb2a562b6b175a165e7e13ddc73ca9b241c927c21" Nov 02 14:30:05.295385 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.295366 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4_odh-ai-gateway-infra(49c1bd66-084f-49ff-87f5-4111ce798e5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" Nov 02 14:30:05.296124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.296107 4431 scope.go:122] "RemoveContainer" containerID="cb77a0cb364b2cbdd6985b4a1ab3c4c9f2c49358e343fc1c57cd89b514f0605f" Nov 02 14:30:05.296966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.296946 4431 generic.go:356] "Generic (PLEG): container finished" podID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerID="9fceb5fb01643afcf2d7ee44f331d0bac762fe3080fe433ef374226d25726e61" exitCode=60 Nov 02 14:30:05.297043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.297019 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" event={"ID":"c1aea439-9e6a-42dd-a70f-e30d47a19f19","Type":"ContainerDied","Data":"9fceb5fb01643afcf2d7ee44f331d0bac762fe3080fe433ef374226d25726e61"} Nov 02 14:30:05.297208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.297192 4431 scope.go:122] "RemoveContainer" containerID="9fceb5fb01643afcf2d7ee44f331d0bac762fe3080fe433ef374226d25726e61" Nov 02 14:30:05.297411 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.297394 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4_odh-ai-gateway-infra(c1aea439-9e6a-42dd-a70f-e30d47a19f19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" Nov 02 14:30:05.299001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.298982 4431 generic.go:356] "Generic (PLEG): container finished" podID="76ef1614-c978-4046-85fe-15c48736d0da" containerID="02e22d58f2eedc261f725019bb49a22b9cbf15cb1afdc22c592b843b612b3d87" exitCode=60 Nov 02 14:30:05.299092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.299006 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" event={"ID":"76ef1614-c978-4046-85fe-15c48736d0da","Type":"ContainerDied","Data":"02e22d58f2eedc261f725019bb49a22b9cbf15cb1afdc22c592b843b612b3d87"} Nov 02 14:30:05.299281 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.299249 4431 scope.go:122] "RemoveContainer" containerID="02e22d58f2eedc261f725019bb49a22b9cbf15cb1afdc22c592b843b612b3d87" Nov 02 14:30:05.299994 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.299966 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx_odh-ai-gateway-infra(76ef1614-c978-4046-85fe-15c48736d0da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" podUID="76ef1614-c978-4046-85fe-15c48736d0da" Nov 02 14:30:05.301151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.301127 4431 generic.go:356] "Generic (PLEG): container finished" podID="fbb841a6-f88a-4139-a862-78938304e503" containerID="53d633eea425be2a3a38f25b2f13b9463a0be00c6fbe49e6a6b74e8be7e6bdc6" exitCode=60 Nov 02 14:30:05.301240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.301201 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" event={"ID":"fbb841a6-f88a-4139-a862-78938304e503","Type":"ContainerDied","Data":"53d633eea425be2a3a38f25b2f13b9463a0be00c6fbe49e6a6b74e8be7e6bdc6"} Nov 02 14:30:05.301420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.301405 4431 scope.go:122] "RemoveContainer" containerID="53d633eea425be2a3a38f25b2f13b9463a0be00c6fbe49e6a6b74e8be7e6bdc6" Nov 02 14:30:05.301624 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.301601 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr_odh-ai-gateway-infra(fbb841a6-f88a-4139-a862-78938304e503)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" podUID="fbb841a6-f88a-4139-a862-78938304e503" Nov 02 14:30:05.306870 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.305662 4431 scope.go:122] "RemoveContainer" containerID="4c7c17df18673edfaea4bb8917c27debf6297cae18ed07edd863e049fa8d6263" Nov 02 14:30:05.306870 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:05.305877 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893830-h5bjw_odh-ai-gateway-infra(6424d0ca-8aa6-4897-a06b-7168cd41016c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" Nov 02 14:30:05.312016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.311996 4431 scope.go:122] "RemoveContainer" containerID="78307d98e68dfcd2b3aa5a2d47a9132bc9010582cfec7c7943144756e2c6a05d" Nov 02 14:30:05.323156 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.323134 4431 scope.go:122] "RemoveContainer" containerID="01895f8fa21bdd272cc3d4e820007e1df0bd13098daa372394d643ca3e7b4c3b" Nov 02 14:30:05.335024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.335000 4431 scope.go:122] "RemoveContainer" containerID="3d96aed95dafa4273488b7aba53c34bfbcf75ed962722e09bc586ffc568717f8" Nov 02 14:30:05.346266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.346235 4431 scope.go:122] "RemoveContainer" containerID="c6619a4b76aeb1fd1a7629eb45978fbfd1e30ef1a1b6457845d945e6caabc39e" Nov 02 14:30:05.358209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.358190 4431 scope.go:122] "RemoveContainer" containerID="a9a9c9ed9d336e6975997c86f13371682e9f732541019306d380bc97c6a4e0c0" Nov 02 14:30:05.371942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.371921 4431 scope.go:122] "RemoveContainer" containerID="547a42382980965aebf5fea0fc4b7fb23cee2c38ee8364e32d78c95bec6f7f52" Nov 02 14:30:05.383540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:05.383504 4431 scope.go:122] "RemoveContainer" containerID="5f45d78227f48e14e97c41fe9cfd2fa3c096fa7b90a6de515ac8844b8ee52f41" Nov 02 14:30:06.314710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.314675 4431 scope.go:122] "RemoveContainer" containerID="406bb317623af6579305afdc14e73c27d1e92f810bad8644f5ba2c6f28b8502e" Nov 02 14:30:06.315325 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.314991 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn_odh-ai-gateway-infra(3fee292c-b279-4973-9b39-2a009a781011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" podUID="3fee292c-b279-4973-9b39-2a009a781011" Nov 02 14:30:06.316308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.316288 4431 scope.go:122] "RemoveContainer" containerID="6266b6cef9ddd66ab6cd029aab2091c515c9d4a679cbe3e86602a340f32db1f4" Nov 02 14:30:06.316601 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.316570 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt_odh-ai-gateway-infra(b5ba8971-616e-43a8-9122-68d990259d85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" podUID="b5ba8971-616e-43a8-9122-68d990259d85" Nov 02 14:30:06.317916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.317897 4431 scope.go:122] "RemoveContainer" containerID="ed70e2f2e532d2dd1e4d6895a85f7e97b6f2abf5105b922289cb74b590869a0d" Nov 02 14:30:06.318175 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.318153 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff_odh-ai-gateway-infra(ce56cca9-81f2-433f-baf5-a61fb490f1f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" Nov 02 14:30:06.319466 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.319450 4431 scope.go:122] "RemoveContainer" containerID="037765bb70775157d8338ea128d2fada689dbbc83fdcb9d78ed2c6328d29c41e" Nov 02 14:30:06.319690 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.319671 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m_odh-ai-gateway-infra(ad7316d0-07ef-4664-9345-d00cd333551d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" Nov 02 14:30:06.320879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.320861 4431 scope.go:122] "RemoveContainer" containerID="bbd3e5b23336a04451c99ac167038a4e3e6d2267c4c2ada29256fc083c323035" Nov 02 14:30:06.321102 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.321081 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp_odh-ai-gateway-infra(cee0b62b-657a-4897-8e74-191f9ecb42d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" Nov 02 14:30:06.322368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.322287 4431 scope.go:122] "RemoveContainer" containerID="739b5e2168c0644b159fb3efb2a562b6b175a165e7e13ddc73ca9b241c927c21" Nov 02 14:30:06.322520 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.322495 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4_odh-ai-gateway-infra(49c1bd66-084f-49ff-87f5-4111ce798e5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" Nov 02 14:30:06.323879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.323860 4431 scope.go:122] "RemoveContainer" containerID="9fceb5fb01643afcf2d7ee44f331d0bac762fe3080fe433ef374226d25726e61" Nov 02 14:30:06.324081 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.324062 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4_odh-ai-gateway-infra(c1aea439-9e6a-42dd-a70f-e30d47a19f19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" Nov 02 14:30:06.325353 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.325336 4431 scope.go:122] "RemoveContainer" containerID="02e22d58f2eedc261f725019bb49a22b9cbf15cb1afdc22c592b843b612b3d87" Nov 02 14:30:06.325542 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.325522 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx_odh-ai-gateway-infra(76ef1614-c978-4046-85fe-15c48736d0da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" podUID="76ef1614-c978-4046-85fe-15c48736d0da" Nov 02 14:30:06.326802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:06.326766 4431 scope.go:122] "RemoveContainer" containerID="53d633eea425be2a3a38f25b2f13b9463a0be00c6fbe49e6a6b74e8be7e6bdc6" Nov 02 14:30:06.326964 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:06.326947 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr_odh-ai-gateway-infra(fbb841a6-f88a-4139-a862-78938304e503)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" podUID="fbb841a6-f88a-4139-a862-78938304e503" Nov 02 14:30:08.316753 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.316702 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8"] Nov 02 14:30:08.325902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.325863 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.371271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.368139 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-928a7a-openshift-default-dockercfg-864hl\"" Nov 02 14:30:08.374307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.374280 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8"] Nov 02 14:30:08.399362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399331 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-credential-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.399487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399368 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-token\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.399533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399507 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-certs\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.399603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399586 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-envoy\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.399646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399623 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxns\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-kube-api-access-dqxns\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.399721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399706 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-data\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.399762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399753 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-podinfo\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.399847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399831 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e0282-f36c-44cf-8a20-af39dcb9302c-istiod-ca-cert\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.399886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.399871 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501412 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-credential-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501449 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-token\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501494 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-certs\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501533 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-envoy\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501559 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxns\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-kube-api-access-dqxns\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501596 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-data\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501635 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-podinfo\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501679 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e0282-f36c-44cf-8a20-af39dcb9302c-istiod-ca-cert\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501706 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501874 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-credential-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.501943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.501901 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-certs\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.502230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.502167 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-data\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.502230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.502184 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-socket\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.502587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.502563 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e0282-f36c-44cf-8a20-af39dcb9302c-istiod-ca-cert\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.504264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.504239 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-envoy\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.504544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.504524 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-podinfo\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.527313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.527261 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-token\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.527412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.527394 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxns\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-kube-api-access-dqxns\") pod \"e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:08.657799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:08.657449 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:09.024512 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:09.024481 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8"] Nov 02 14:30:09.025920 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:09.025894 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04e0282_f36c_44cf_8a20_af39dcb9302c.slice/crio-6419b338d60a7f531d2e35c22af486a0a4f53b07ab018a66f4ff8645ce0bb0f5 WatchSource:0}: Error finding container 6419b338d60a7f531d2e35c22af486a0a4f53b07ab018a66f4ff8645ce0bb0f5: Status 404 returned error can't find the container with id 6419b338d60a7f531d2e35c22af486a0a4f53b07ab018a66f4ff8645ce0bb0f5 Nov 02 14:30:09.029088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:09.029056 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:09.029173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:09.029126 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:09.029173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:09.029159 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:09.355726 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:09.355639 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" event={"ID":"c04e0282-f36c-44cf-8a20-af39dcb9302c","Type":"ContainerStarted","Data":"ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692"} Nov 02 14:30:09.356221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:09.355732 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" event={"ID":"c04e0282-f36c-44cf-8a20-af39dcb9302c","Type":"ContainerStarted","Data":"6419b338d60a7f531d2e35c22af486a0a4f53b07ab018a66f4ff8645ce0bb0f5"} Nov 02 14:30:09.405099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:09.405049 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" podStartSLOduration=1.405034768 podStartE2EDuration="1.405034768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:09.403602424 +0000 UTC m=+1974.440707357" watchObservedRunningTime="2026-11-02 14:30:09.405034768 +0000 UTC m=+1974.442139699" Nov 02 14:30:09.658451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:09.658422 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:10.663381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:10.663346 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:11.369764 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:11.369732 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:11.370811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:11.370765 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:30:13.039722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:13.039688 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc"] Nov 02 14:30:13.055465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:13.055442 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe456d6-ncmcc"] Nov 02 14:30:13.076719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:13.076689 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cbd4eb-71ef-4fd8-83e3-7538c2606205" path="/var/lib/kubelet/pods/e4cbd4eb-71ef-4fd8-83e3-7538c2606205/volumes" Nov 02 14:30:15.198648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.198610 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn"] Nov 02 14:30:15.204457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.204433 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.235429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.235403 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-778b2bce-openshift-default-dockercfg-jd4nq\"" Nov 02 14:30:15.239388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.239364 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn"] Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.327705 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-token\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.327835 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-socket\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.327883 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istiod-ca-cert\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.327931 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-podinfo\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.327981 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-envoy\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.328043 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-data\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.328078 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-certs\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.328172 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xl4\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-kube-api-access-v7xl4\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.333800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.328223 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-credential-socket\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429436 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-token\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429535 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-socket\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429573 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istiod-ca-cert\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429616 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-podinfo\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429657 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-envoy\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429713 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-data\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429754 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-certs\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429865 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xl4\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-kube-api-access-v7xl4\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.429972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.429915 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-credential-socket\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.430433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.430406 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-credential-socket\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.430559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.430485 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-socket\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.430948 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.430921 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-certs\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.431217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.431201 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istiod-ca-cert\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.431331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.431218 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-data\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.433303 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.433279 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-podinfo\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.435014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.434995 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-envoy\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.455529 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.455460 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-token\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.456266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.456241 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xl4\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-kube-api-access-v7xl4\") pod \"e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.520237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.520207 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:15.853177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.853096 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn"] Nov 02 14:30:15.864548 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:15.864509 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d580e7_f40b_47ea_b707_e1c6c3e144ac.slice/crio-7b6498866bc6faec4d4b8e17de0644d1e7c092035fdf9e10266f71819d793e77 WatchSource:0}: Error finding container 7b6498866bc6faec4d4b8e17de0644d1e7c092035fdf9e10266f71819d793e77: Status 404 returned error can't find the container with id 7b6498866bc6faec4d4b8e17de0644d1e7c092035fdf9e10266f71819d793e77 Nov 02 14:30:15.867877 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.867840 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:15.867996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.867928 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:15.867996 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:15.867980 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:16.049804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.049751 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:30:16.068817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.068516 4431 scope.go:122] "RemoveContainer" containerID="406bb317623af6579305afdc14e73c27d1e92f810bad8644f5ba2c6f28b8502e" Nov 02 14:30:16.068817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.068659 4431 scope.go:122] "RemoveContainer" containerID="4c7c17df18673edfaea4bb8917c27debf6297cae18ed07edd863e049fa8d6263" Nov 02 14:30:16.069338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.069187 4431 scope.go:122] "RemoveContainer" containerID="53d633eea425be2a3a38f25b2f13b9463a0be00c6fbe49e6a6b74e8be7e6bdc6" Nov 02 14:30:16.411137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.411108 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" event={"ID":"3fee292c-b279-4973-9b39-2a009a781011","Type":"ContainerStarted","Data":"c2c85ce0999dd2cf3320ac88f80999a24d83b4a18b0c1e726beed97992826570"} Nov 02 14:30:16.412927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.412905 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" event={"ID":"61d580e7-f40b-47ea-b707-e1c6c3e144ac","Type":"ContainerStarted","Data":"1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376"} Nov 02 14:30:16.413023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.412952 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" event={"ID":"61d580e7-f40b-47ea-b707-e1c6c3e144ac","Type":"ContainerStarted","Data":"7b6498866bc6faec4d4b8e17de0644d1e7c092035fdf9e10266f71819d793e77"} Nov 02 14:30:16.510749 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.510703 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" podStartSLOduration=1.5106895150000001 podStartE2EDuration="1.510689515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:16.507041587 +0000 UTC m=+1981.544146532" watchObservedRunningTime="2026-11-02 14:30:16.510689515 +0000 UTC m=+1981.547794447" Nov 02 14:30:16.520922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.520897 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:16.527665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:16.527647 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:17.068642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.068606 4431 scope.go:122] "RemoveContainer" containerID="6266b6cef9ddd66ab6cd029aab2091c515c9d4a679cbe3e86602a340f32db1f4" Nov 02 14:30:17.068754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.068662 4431 scope.go:122] "RemoveContainer" containerID="c042458cbfe36b61fee29b113c559c3affea12611f33dc9e59f35a3188daee82" Nov 02 14:30:17.422739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.422707 4431 generic.go:356] "Generic (PLEG): container finished" podID="3fee292c-b279-4973-9b39-2a009a781011" containerID="c2c85ce0999dd2cf3320ac88f80999a24d83b4a18b0c1e726beed97992826570" exitCode=60 Nov 02 14:30:17.423200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.422806 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" event={"ID":"3fee292c-b279-4973-9b39-2a009a781011","Type":"ContainerDied","Data":"c2c85ce0999dd2cf3320ac88f80999a24d83b4a18b0c1e726beed97992826570"} Nov 02 14:30:17.423200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.422866 4431 scope.go:122] "RemoveContainer" containerID="406bb317623af6579305afdc14e73c27d1e92f810bad8644f5ba2c6f28b8502e" Nov 02 14:30:17.423312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.423297 4431 scope.go:122] "RemoveContainer" containerID="c2c85ce0999dd2cf3320ac88f80999a24d83b4a18b0c1e726beed97992826570" Nov 02 14:30:17.423680 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:17.423651 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn_odh-ai-gateway-infra(3fee292c-b279-4973-9b39-2a009a781011)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" podUID="3fee292c-b279-4973-9b39-2a009a781011" Nov 02 14:30:17.425434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.425410 4431 generic.go:356] "Generic (PLEG): container finished" podID="fbb841a6-f88a-4139-a862-78938304e503" containerID="e55f76d112a4b3696bcbb2b3c6eacdca128304a79837e552bd6c964756bdc19c" exitCode=60 Nov 02 14:30:17.425548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.425476 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" event={"ID":"fbb841a6-f88a-4139-a862-78938304e503","Type":"ContainerDied","Data":"e55f76d112a4b3696bcbb2b3c6eacdca128304a79837e552bd6c964756bdc19c"} Nov 02 14:30:17.425916 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.425897 4431 scope.go:122] "RemoveContainer" containerID="e55f76d112a4b3696bcbb2b3c6eacdca128304a79837e552bd6c964756bdc19c" Nov 02 14:30:17.426248 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:17.426228 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr_odh-ai-gateway-infra(fbb841a6-f88a-4139-a862-78938304e503)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" podUID="fbb841a6-f88a-4139-a862-78938304e503" Nov 02 14:30:17.427640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.427621 4431 generic.go:356] "Generic (PLEG): container finished" podID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerID="1da332b0a77ef530c4ca2b881fdc3ff39933ba0319b5c01e6be6f5c1a43ce215" exitCode=60 Nov 02 14:30:17.427748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.427694 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" event={"ID":"6424d0ca-8aa6-4897-a06b-7168cd41016c","Type":"ContainerDied","Data":"1da332b0a77ef530c4ca2b881fdc3ff39933ba0319b5c01e6be6f5c1a43ce215"} Nov 02 14:30:17.427956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.427930 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:17.428317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.428292 4431 scope.go:122] "RemoveContainer" containerID="1da332b0a77ef530c4ca2b881fdc3ff39933ba0319b5c01e6be6f5c1a43ce215" Nov 02 14:30:17.428612 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:17.428564 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893830-h5bjw_odh-ai-gateway-infra(6424d0ca-8aa6-4897-a06b-7168cd41016c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" Nov 02 14:30:17.428986 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.428970 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:30:17.438081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.438058 4431 scope.go:122] "RemoveContainer" containerID="53d633eea425be2a3a38f25b2f13b9463a0be00c6fbe49e6a6b74e8be7e6bdc6" Nov 02 14:30:17.451209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.451189 4431 scope.go:122] "RemoveContainer" containerID="4c7c17df18673edfaea4bb8917c27debf6297cae18ed07edd863e049fa8d6263" Nov 02 14:30:17.469801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.469765 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn"] Nov 02 14:30:17.623923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:17.623846 4431 scope.go:122] "RemoveContainer" containerID="a0386251bc3c2cbf67167929c0584ee476919476eefa453a2cd5341254a60bca" Nov 02 14:30:18.069439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.069410 4431 scope.go:122] "RemoveContainer" containerID="9fceb5fb01643afcf2d7ee44f331d0bac762fe3080fe433ef374226d25726e61" Nov 02 14:30:18.069650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.069515 4431 scope.go:122] "RemoveContainer" containerID="037765bb70775157d8338ea128d2fada689dbbc83fdcb9d78ed2c6328d29c41e" Nov 02 14:30:18.069650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.069571 4431 scope.go:122] "RemoveContainer" containerID="bbd3e5b23336a04451c99ac167038a4e3e6d2267c4c2ada29256fc083c323035" Nov 02 14:30:18.069800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.069663 4431 scope.go:122] "RemoveContainer" containerID="ed70e2f2e532d2dd1e4d6895a85f7e97b6f2abf5105b922289cb74b590869a0d" Nov 02 14:30:18.441947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.441919 4431 generic.go:356] "Generic (PLEG): container finished" podID="54b19435-f528-48b4-b807-70e9c6c80abf" containerID="62a61805e15db7ac0fee95817e98364443acfa6a0de10783c7d3d14edcddd32e" exitCode=60 Nov 02 14:30:18.442313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.441985 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" event={"ID":"54b19435-f528-48b4-b807-70e9c6c80abf","Type":"ContainerDied","Data":"62a61805e15db7ac0fee95817e98364443acfa6a0de10783c7d3d14edcddd32e"} Nov 02 14:30:18.442313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.442038 4431 scope.go:122] "RemoveContainer" containerID="c042458cbfe36b61fee29b113c559c3affea12611f33dc9e59f35a3188daee82" Nov 02 14:30:18.442451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.442430 4431 scope.go:122] "RemoveContainer" containerID="62a61805e15db7ac0fee95817e98364443acfa6a0de10783c7d3d14edcddd32e" Nov 02 14:30:18.442807 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:18.442761 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6_odh-ai-gateway-infra(54b19435-f528-48b4-b807-70e9c6c80abf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" Nov 02 14:30:18.444309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.444279 4431 generic.go:356] "Generic (PLEG): container finished" podID="b5ba8971-616e-43a8-9122-68d990259d85" containerID="03fa4f32642559435313bab0f23e4af1a9d90e2d580442108319898b6f010a9d" exitCode=60 Nov 02 14:30:18.444422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.444351 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" event={"ID":"b5ba8971-616e-43a8-9122-68d990259d85","Type":"ContainerDied","Data":"03fa4f32642559435313bab0f23e4af1a9d90e2d580442108319898b6f010a9d"} Nov 02 14:30:18.444808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.444764 4431 scope.go:122] "RemoveContainer" containerID="03fa4f32642559435313bab0f23e4af1a9d90e2d580442108319898b6f010a9d" Nov 02 14:30:18.445131 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:18.445111 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt_odh-ai-gateway-infra(b5ba8971-616e-43a8-9122-68d990259d85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" podUID="b5ba8971-616e-43a8-9122-68d990259d85" Nov 02 14:30:18.456705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.456689 4431 scope.go:122] "RemoveContainer" containerID="6266b6cef9ddd66ab6cd029aab2091c515c9d4a679cbe3e86602a340f32db1f4" Nov 02 14:30:18.558063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.557647 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw"] Nov 02 14:30:18.591294 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.591266 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr"] Nov 02 14:30:18.623689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.623664 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:18.731005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.730890 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:18.789813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.788990 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:18.789813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.789592 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fee292c-b279-4973-9b39-2a009a781011-kube-api-access-hmbg6\" (UniqueName: \"kubernetes.io/projected/3fee292c-b279-4973-9b39-2a009a781011-kube-api-access-hmbg6\") pod \"3fee292c-b279-4973-9b39-2a009a781011\" (UID: \"3fee292c-b279-4973-9b39-2a009a781011\") " Nov 02 14:30:18.790040 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.789632 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fee292c-b279-4973-9b39-2a009a781011-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fee292c-b279-4973-9b39-2a009a781011-maas-api-service-ca\") pod \"3fee292c-b279-4973-9b39-2a009a781011\" (UID: \"3fee292c-b279-4973-9b39-2a009a781011\") " Nov 02 14:30:18.790106 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.789970 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fee292c-b279-4973-9b39-2a009a781011-maas-api-service-ca" pod "3fee292c-b279-4973-9b39-2a009a781011" (UID: "3fee292c-b279-4973-9b39-2a009a781011"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:18.790690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.790672 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fee292c-b279-4973-9b39-2a009a781011-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:18.793395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.793372 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fee292c-b279-4973-9b39-2a009a781011-kube-api-access-hmbg6" pod "3fee292c-b279-4973-9b39-2a009a781011" (UID: "3fee292c-b279-4973-9b39-2a009a781011"). InnerVolumeSpecName "kube-api-access-hmbg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:18.892120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.892097 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb841a6-f88a-4139-a862-78938304e503-kube-api-access-2qp6l\" (UniqueName: \"kubernetes.io/projected/fbb841a6-f88a-4139-a862-78938304e503-kube-api-access-2qp6l\") pod \"fbb841a6-f88a-4139-a862-78938304e503\" (UID: \"fbb841a6-f88a-4139-a862-78938304e503\") " Nov 02 14:30:18.892263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.892194 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6424d0ca-8aa6-4897-a06b-7168cd41016c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6424d0ca-8aa6-4897-a06b-7168cd41016c-maas-api-service-ca\") pod \"6424d0ca-8aa6-4897-a06b-7168cd41016c\" (UID: \"6424d0ca-8aa6-4897-a06b-7168cd41016c\") " Nov 02 14:30:18.892314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.892257 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6424d0ca-8aa6-4897-a06b-7168cd41016c-kube-api-access-gfdcd\" (UniqueName: \"kubernetes.io/projected/6424d0ca-8aa6-4897-a06b-7168cd41016c-kube-api-access-gfdcd\") pod \"6424d0ca-8aa6-4897-a06b-7168cd41016c\" (UID: \"6424d0ca-8aa6-4897-a06b-7168cd41016c\") " Nov 02 14:30:18.892455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.892433 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb841a6-f88a-4139-a862-78938304e503-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb841a6-f88a-4139-a862-78938304e503-maas-api-service-ca\") pod \"fbb841a6-f88a-4139-a862-78938304e503\" (UID: \"fbb841a6-f88a-4139-a862-78938304e503\") " Nov 02 14:30:18.892584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.892553 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6424d0ca-8aa6-4897-a06b-7168cd41016c-maas-api-service-ca" pod "6424d0ca-8aa6-4897-a06b-7168cd41016c" (UID: "6424d0ca-8aa6-4897-a06b-7168cd41016c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:18.892751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.892732 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb841a6-f88a-4139-a862-78938304e503-maas-api-service-ca" pod "fbb841a6-f88a-4139-a862-78938304e503" (UID: "fbb841a6-f88a-4139-a862-78938304e503"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:18.893120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.893100 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmbg6\" (UniqueName: \"kubernetes.io/projected/3fee292c-b279-4973-9b39-2a009a781011-kube-api-access-hmbg6\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:18.893200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.893121 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb841a6-f88a-4139-a862-78938304e503-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:18.893200 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.893131 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6424d0ca-8aa6-4897-a06b-7168cd41016c-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:18.894142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.894118 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb841a6-f88a-4139-a862-78938304e503-kube-api-access-2qp6l" pod "fbb841a6-f88a-4139-a862-78938304e503" (UID: "fbb841a6-f88a-4139-a862-78938304e503"). InnerVolumeSpecName "kube-api-access-2qp6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:18.894281 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.894264 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6424d0ca-8aa6-4897-a06b-7168cd41016c-kube-api-access-gfdcd" pod "6424d0ca-8aa6-4897-a06b-7168cd41016c" (UID: "6424d0ca-8aa6-4897-a06b-7168cd41016c"). InnerVolumeSpecName "kube-api-access-gfdcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:18.994745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.994707 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qp6l\" (UniqueName: \"kubernetes.io/projected/fbb841a6-f88a-4139-a862-78938304e503-kube-api-access-2qp6l\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:18.994745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:18.994744 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfdcd\" (UniqueName: \"kubernetes.io/projected/6424d0ca-8aa6-4897-a06b-7168cd41016c-kube-api-access-gfdcd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:19.068691 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.068661 4431 scope.go:122] "RemoveContainer" containerID="739b5e2168c0644b159fb3efb2a562b6b175a165e7e13ddc73ca9b241c927c21" Nov 02 14:30:19.069014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.068993 4431 scope.go:122] "RemoveContainer" containerID="02e22d58f2eedc261f725019bb49a22b9cbf15cb1afdc22c592b843b612b3d87" Nov 02 14:30:19.456109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.456071 4431 generic.go:356] "Generic (PLEG): container finished" podID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerID="d75c09eca2edc958758b8251688b3cd1ff672936be923413ad0257dcb60c4ade" exitCode=60 Nov 02 14:30:19.456473 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.456138 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" event={"ID":"ce56cca9-81f2-433f-baf5-a61fb490f1f5","Type":"ContainerDied","Data":"d75c09eca2edc958758b8251688b3cd1ff672936be923413ad0257dcb60c4ade"} Nov 02 14:30:19.456473 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.456211 4431 scope.go:122] "RemoveContainer" containerID="ed70e2f2e532d2dd1e4d6895a85f7e97b6f2abf5105b922289cb74b590869a0d" Nov 02 14:30:19.456610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.456596 4431 scope.go:122] "RemoveContainer" containerID="d75c09eca2edc958758b8251688b3cd1ff672936be923413ad0257dcb60c4ade" Nov 02 14:30:19.456945 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:19.456920 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff_odh-ai-gateway-infra(ce56cca9-81f2-433f-baf5-a61fb490f1f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" Nov 02 14:30:19.458379 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.458357 4431 generic.go:356] "Generic (PLEG): container finished" podID="ad7316d0-07ef-4664-9345-d00cd333551d" containerID="cf719991ecc87683c2d9fbf5781ff52a27ef31c793f0030929b63e8ba986cd10" exitCode=60 Nov 02 14:30:19.458486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.458412 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" event={"ID":"ad7316d0-07ef-4664-9345-d00cd333551d","Type":"ContainerDied","Data":"cf719991ecc87683c2d9fbf5781ff52a27ef31c793f0030929b63e8ba986cd10"} Nov 02 14:30:19.458849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.458832 4431 scope.go:122] "RemoveContainer" containerID="cf719991ecc87683c2d9fbf5781ff52a27ef31c793f0030929b63e8ba986cd10" Nov 02 14:30:19.459145 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:19.459119 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m_odh-ai-gateway-infra(ad7316d0-07ef-4664-9345-d00cd333551d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" Nov 02 14:30:19.460376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.460358 4431 generic.go:356] "Generic (PLEG): container finished" podID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerID="54a49f5e6b729988f9619b14c241c7478932503e86a4a06d0023d460e66520a7" exitCode=60 Nov 02 14:30:19.460480 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.460432 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" event={"ID":"cee0b62b-657a-4897-8e74-191f9ecb42d6","Type":"ContainerDied","Data":"54a49f5e6b729988f9619b14c241c7478932503e86a4a06d0023d460e66520a7"} Nov 02 14:30:19.460810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.460787 4431 scope.go:122] "RemoveContainer" containerID="54a49f5e6b729988f9619b14c241c7478932503e86a4a06d0023d460e66520a7" Nov 02 14:30:19.461074 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:19.461054 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp_odh-ai-gateway-infra(cee0b62b-657a-4897-8e74-191f9ecb42d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" Nov 02 14:30:19.462458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.462440 4431 generic.go:356] "Generic (PLEG): container finished" podID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerID="511b71b231f08f7fd7972a13f9627ef9ef825ab243aa90174d29e38137c92943" exitCode=60 Nov 02 14:30:19.462543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.462511 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" event={"ID":"c1aea439-9e6a-42dd-a70f-e30d47a19f19","Type":"ContainerDied","Data":"511b71b231f08f7fd7972a13f9627ef9ef825ab243aa90174d29e38137c92943"} Nov 02 14:30:19.462872 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.462857 4431 scope.go:122] "RemoveContainer" containerID="511b71b231f08f7fd7972a13f9627ef9ef825ab243aa90174d29e38137c92943" Nov 02 14:30:19.463074 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:19.463057 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4_odh-ai-gateway-infra(c1aea439-9e6a-42dd-a70f-e30d47a19f19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" Nov 02 14:30:19.463835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.463821 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" Nov 02 14:30:19.463921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.463820 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr" event={"ID":"fbb841a6-f88a-4139-a862-78938304e503","Type":"ContainerDied","Data":"b189d23b4332f8b5b5f7691b568288f60327234637e70a359f8ca7e8a208e4dc"} Nov 02 14:30:19.465277 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.465259 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" Nov 02 14:30:19.465381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.465329 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw" event={"ID":"6424d0ca-8aa6-4897-a06b-7168cd41016c","Type":"ContainerDied","Data":"d5f29d0fc2fd7f9ce2035c88f8f97879cdd45c13268bdd67092fc891e5e8f571"} Nov 02 14:30:19.466813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.466792 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" event={"ID":"3fee292c-b279-4973-9b39-2a009a781011","Type":"ContainerDied","Data":"5dc771c01656fe9106654559cff4837c048da8f9ebe352b8f7e0da82e9eb3faf"} Nov 02 14:30:19.466909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.466863 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn" Nov 02 14:30:19.470591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.470562 4431 scope.go:122] "RemoveContainer" containerID="037765bb70775157d8338ea128d2fada689dbbc83fdcb9d78ed2c6328d29c41e" Nov 02 14:30:19.487505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.487371 4431 scope.go:122] "RemoveContainer" containerID="bbd3e5b23336a04451c99ac167038a4e3e6d2267c4c2ada29256fc083c323035" Nov 02 14:30:19.499869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.499850 4431 scope.go:122] "RemoveContainer" containerID="9fceb5fb01643afcf2d7ee44f331d0bac762fe3080fe433ef374226d25726e61" Nov 02 14:30:19.516972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.516954 4431 scope.go:122] "RemoveContainer" containerID="e55f76d112a4b3696bcbb2b3c6eacdca128304a79837e552bd6c964756bdc19c" Nov 02 14:30:19.526299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.526282 4431 scope.go:122] "RemoveContainer" containerID="1da332b0a77ef530c4ca2b881fdc3ff39933ba0319b5c01e6be6f5c1a43ce215" Nov 02 14:30:19.535000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.534983 4431 scope.go:122] "RemoveContainer" containerID="c2c85ce0999dd2cf3320ac88f80999a24d83b4a18b0c1e726beed97992826570" Nov 02 14:30:19.545582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.545559 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt"] Nov 02 14:30:19.550654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.550633 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw"] Nov 02 14:30:19.565821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.564939 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893830-h5bjw"] Nov 02 14:30:19.607166 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.606931 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6"] Nov 02 14:30:19.667491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.667014 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr"] Nov 02 14:30:19.679802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.678413 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4be645-29893830-f4lbr"] Nov 02 14:30:19.761201 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.761180 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:19.772451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.772424 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn"] Nov 02 14:30:19.777261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.777237 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-866159-29893830-wd9gn"] Nov 02 14:30:19.788972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.788954 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:19.905986 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.905957 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ba8971-616e-43a8-9122-68d990259d85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ba8971-616e-43a8-9122-68d990259d85-maas-api-service-ca\") pod \"b5ba8971-616e-43a8-9122-68d990259d85\" (UID: \"b5ba8971-616e-43a8-9122-68d990259d85\") " Nov 02 14:30:19.906174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.906052 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ba8971-616e-43a8-9122-68d990259d85-kube-api-access-st7gw\" (UniqueName: \"kubernetes.io/projected/b5ba8971-616e-43a8-9122-68d990259d85-kube-api-access-st7gw\") pod \"b5ba8971-616e-43a8-9122-68d990259d85\" (UID: \"b5ba8971-616e-43a8-9122-68d990259d85\") " Nov 02 14:30:19.906237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.906181 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b19435-f528-48b4-b807-70e9c6c80abf-kube-api-access-pt2sx\" (UniqueName: \"kubernetes.io/projected/54b19435-f528-48b4-b807-70e9c6c80abf-kube-api-access-pt2sx\") pod \"54b19435-f528-48b4-b807-70e9c6c80abf\" (UID: \"54b19435-f528-48b4-b807-70e9c6c80abf\") " Nov 02 14:30:19.906302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.906285 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b19435-f528-48b4-b807-70e9c6c80abf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b19435-f528-48b4-b807-70e9c6c80abf-maas-api-service-ca\") pod \"54b19435-f528-48b4-b807-70e9c6c80abf\" (UID: \"54b19435-f528-48b4-b807-70e9c6c80abf\") " Nov 02 14:30:19.907047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.907020 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b19435-f528-48b4-b807-70e9c6c80abf-maas-api-service-ca" pod "54b19435-f528-48b4-b807-70e9c6c80abf" (UID: "54b19435-f528-48b4-b807-70e9c6c80abf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:19.907499 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.907471 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ba8971-616e-43a8-9122-68d990259d85-maas-api-service-ca" pod "b5ba8971-616e-43a8-9122-68d990259d85" (UID: "b5ba8971-616e-43a8-9122-68d990259d85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:19.909418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.909389 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b19435-f528-48b4-b807-70e9c6c80abf-kube-api-access-pt2sx" pod "54b19435-f528-48b4-b807-70e9c6c80abf" (UID: "54b19435-f528-48b4-b807-70e9c6c80abf"). InnerVolumeSpecName "kube-api-access-pt2sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:19.909818 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:19.909769 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ba8971-616e-43a8-9122-68d990259d85-kube-api-access-st7gw" pod "b5ba8971-616e-43a8-9122-68d990259d85" (UID: "b5ba8971-616e-43a8-9122-68d990259d85"). InnerVolumeSpecName "kube-api-access-st7gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:20.007905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.007836 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54b19435-f528-48b4-b807-70e9c6c80abf-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.007905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.007860 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ba8971-616e-43a8-9122-68d990259d85-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.007905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.007871 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7gw\" (UniqueName: \"kubernetes.io/projected/b5ba8971-616e-43a8-9122-68d990259d85-kube-api-access-st7gw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.007905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.007879 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2sx\" (UniqueName: \"kubernetes.io/projected/54b19435-f528-48b4-b807-70e9c6c80abf-kube-api-access-pt2sx\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.476284 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.476250 4431 generic.go:356] "Generic (PLEG): container finished" podID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerID="3d8333d1a87a9ec3134616ccdfc0f74258445226029b179191c0008e624166a0" exitCode=60 Nov 02 14:30:20.476706 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.476294 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" event={"ID":"49c1bd66-084f-49ff-87f5-4111ce798e5b","Type":"ContainerDied","Data":"3d8333d1a87a9ec3134616ccdfc0f74258445226029b179191c0008e624166a0"} Nov 02 14:30:20.476706 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.476358 4431 scope.go:122] "RemoveContainer" containerID="739b5e2168c0644b159fb3efb2a562b6b175a165e7e13ddc73ca9b241c927c21" Nov 02 14:30:20.476874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.476858 4431 scope.go:122] "RemoveContainer" containerID="3d8333d1a87a9ec3134616ccdfc0f74258445226029b179191c0008e624166a0" Nov 02 14:30:20.477231 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:20.477207 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4_odh-ai-gateway-infra(49c1bd66-084f-49ff-87f5-4111ce798e5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" Nov 02 14:30:20.480046 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.480026 4431 generic.go:356] "Generic (PLEG): container finished" podID="76ef1614-c978-4046-85fe-15c48736d0da" containerID="fb40bd8263a9991a05a01644359636883d5707fa76d6811cfec4c906b6435dd0" exitCode=60 Nov 02 14:30:20.480131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.480098 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" event={"ID":"76ef1614-c978-4046-85fe-15c48736d0da","Type":"ContainerDied","Data":"fb40bd8263a9991a05a01644359636883d5707fa76d6811cfec4c906b6435dd0"} Nov 02 14:30:20.480467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.480446 4431 scope.go:122] "RemoveContainer" containerID="fb40bd8263a9991a05a01644359636883d5707fa76d6811cfec4c906b6435dd0" Nov 02 14:30:20.480788 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:20.480749 4431 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx_odh-ai-gateway-infra(76ef1614-c978-4046-85fe-15c48736d0da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" podUID="76ef1614-c978-4046-85fe-15c48736d0da" Nov 02 14:30:20.484492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.484474 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" Nov 02 14:30:20.484627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.484469 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6" event={"ID":"54b19435-f528-48b4-b807-70e9c6c80abf","Type":"ContainerDied","Data":"40fb456f901da1a1b1f65973c89ed72d8c69b01221cb54365a57b40cce36d6f8"} Nov 02 14:30:20.486058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.486041 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" Nov 02 14:30:20.486164 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.486038 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt" event={"ID":"b5ba8971-616e-43a8-9122-68d990259d85","Type":"ContainerDied","Data":"77afa060bebc945b8683a79017498684fb0134bc7c6b121e8357d2cf09ad2420"} Nov 02 14:30:20.491122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.491104 4431 scope.go:122] "RemoveContainer" containerID="02e22d58f2eedc261f725019bb49a22b9cbf15cb1afdc22c592b843b612b3d87" Nov 02 14:30:20.504561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.504535 4431 scope.go:122] "RemoveContainer" containerID="62a61805e15db7ac0fee95817e98364443acfa6a0de10783c7d3d14edcddd32e" Nov 02 14:30:20.515812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.515766 4431 scope.go:122] "RemoveContainer" containerID="03fa4f32642559435313bab0f23e4af1a9d90e2d580442108319898b6f010a9d" Nov 02 14:30:20.516384 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.516361 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff"] Nov 02 14:30:20.629618 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.629586 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m"] Nov 02 14:30:20.664896 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.664870 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt"] Nov 02 14:30:20.669230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.669207 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f72bc6-29893830-9f8jt"] Nov 02 14:30:20.672194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.672167 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:20.707554 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.707518 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6"] Nov 02 14:30:20.714380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.714354 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9ba664-29893830-kncn6"] Nov 02 14:30:20.716925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.716269 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce56cca9-81f2-433f-baf5-a61fb490f1f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce56cca9-81f2-433f-baf5-a61fb490f1f5-maas-api-service-ca\") pod \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\" (UID: \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\") " Nov 02 14:30:20.716925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.716393 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce56cca9-81f2-433f-baf5-a61fb490f1f5-kube-api-access-2q4z7\" (UniqueName: \"kubernetes.io/projected/ce56cca9-81f2-433f-baf5-a61fb490f1f5-kube-api-access-2q4z7\") pod \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\" (UID: \"ce56cca9-81f2-433f-baf5-a61fb490f1f5\") " Nov 02 14:30:20.717147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.717071 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce56cca9-81f2-433f-baf5-a61fb490f1f5-maas-api-service-ca" pod "ce56cca9-81f2-433f-baf5-a61fb490f1f5" (UID: "ce56cca9-81f2-433f-baf5-a61fb490f1f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:20.719962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.719935 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce56cca9-81f2-433f-baf5-a61fb490f1f5-kube-api-access-2q4z7" pod "ce56cca9-81f2-433f-baf5-a61fb490f1f5" (UID: "ce56cca9-81f2-433f-baf5-a61fb490f1f5"). InnerVolumeSpecName "kube-api-access-2q4z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:20.725630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.725606 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4"] Nov 02 14:30:20.773373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.773344 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:20.817234 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.817192 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7316d0-07ef-4664-9345-d00cd333551d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7316d0-07ef-4664-9345-d00cd333551d-maas-api-service-ca\") pod \"ad7316d0-07ef-4664-9345-d00cd333551d\" (UID: \"ad7316d0-07ef-4664-9345-d00cd333551d\") " Nov 02 14:30:20.817404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.817342 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7316d0-07ef-4664-9345-d00cd333551d-kube-api-access-44hf4\" (UniqueName: \"kubernetes.io/projected/ad7316d0-07ef-4664-9345-d00cd333551d-kube-api-access-44hf4\") pod \"ad7316d0-07ef-4664-9345-d00cd333551d\" (UID: \"ad7316d0-07ef-4664-9345-d00cd333551d\") " Nov 02 14:30:20.817547 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.817524 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7316d0-07ef-4664-9345-d00cd333551d-maas-api-service-ca" pod "ad7316d0-07ef-4664-9345-d00cd333551d" (UID: "ad7316d0-07ef-4664-9345-d00cd333551d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:20.818008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.817987 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4z7\" (UniqueName: \"kubernetes.io/projected/ce56cca9-81f2-433f-baf5-a61fb490f1f5-kube-api-access-2q4z7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.818084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.818015 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce56cca9-81f2-433f-baf5-a61fb490f1f5-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.818084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.818032 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7316d0-07ef-4664-9345-d00cd333551d-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.819709 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.819678 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7316d0-07ef-4664-9345-d00cd333551d-kube-api-access-44hf4" pod "ad7316d0-07ef-4664-9345-d00cd333551d" (UID: "ad7316d0-07ef-4664-9345-d00cd333551d"). InnerVolumeSpecName "kube-api-access-44hf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:20.840865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.840825 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp"] Nov 02 14:30:20.887489 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.887462 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:20.926168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.926133 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1aea439-9e6a-42dd-a70f-e30d47a19f19-kube-api-access-8p8wr\" (UniqueName: \"kubernetes.io/projected/c1aea439-9e6a-42dd-a70f-e30d47a19f19-kube-api-access-8p8wr\") pod \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\" (UID: \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\") " Nov 02 14:30:20.926323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.926215 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1aea439-9e6a-42dd-a70f-e30d47a19f19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1aea439-9e6a-42dd-a70f-e30d47a19f19-maas-api-service-ca\") pod \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\" (UID: \"c1aea439-9e6a-42dd-a70f-e30d47a19f19\") " Nov 02 14:30:20.926808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.926765 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44hf4\" (UniqueName: \"kubernetes.io/projected/ad7316d0-07ef-4664-9345-d00cd333551d-kube-api-access-44hf4\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:20.926995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.926919 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1aea439-9e6a-42dd-a70f-e30d47a19f19-maas-api-service-ca" pod "c1aea439-9e6a-42dd-a70f-e30d47a19f19" (UID: "c1aea439-9e6a-42dd-a70f-e30d47a19f19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:20.929010 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:20.928520 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1aea439-9e6a-42dd-a70f-e30d47a19f19-kube-api-access-8p8wr" pod "c1aea439-9e6a-42dd-a70f-e30d47a19f19" (UID: "c1aea439-9e6a-42dd-a70f-e30d47a19f19"). InnerVolumeSpecName "kube-api-access-8p8wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:21.004300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.004274 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:21.027636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.027593 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee0b62b-657a-4897-8e74-191f9ecb42d6-kube-api-access-7lnvv\" (UniqueName: \"kubernetes.io/projected/cee0b62b-657a-4897-8e74-191f9ecb42d6-kube-api-access-7lnvv\") pod \"cee0b62b-657a-4897-8e74-191f9ecb42d6\" (UID: \"cee0b62b-657a-4897-8e74-191f9ecb42d6\") " Nov 02 14:30:21.027921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.027665 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee0b62b-657a-4897-8e74-191f9ecb42d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee0b62b-657a-4897-8e74-191f9ecb42d6-maas-api-service-ca\") pod \"cee0b62b-657a-4897-8e74-191f9ecb42d6\" (UID: \"cee0b62b-657a-4897-8e74-191f9ecb42d6\") " Nov 02 14:30:21.028500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.028066 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p8wr\" (UniqueName: \"kubernetes.io/projected/c1aea439-9e6a-42dd-a70f-e30d47a19f19-kube-api-access-8p8wr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.028500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.028091 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1aea439-9e6a-42dd-a70f-e30d47a19f19-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.028500 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.028183 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee0b62b-657a-4897-8e74-191f9ecb42d6-maas-api-service-ca" pod "cee0b62b-657a-4897-8e74-191f9ecb42d6" (UID: "cee0b62b-657a-4897-8e74-191f9ecb42d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:21.029995 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.029967 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee0b62b-657a-4897-8e74-191f9ecb42d6-kube-api-access-7lnvv" pod "cee0b62b-657a-4897-8e74-191f9ecb42d6" (UID: "cee0b62b-657a-4897-8e74-191f9ecb42d6"). InnerVolumeSpecName "kube-api-access-7lnvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:21.078940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.078857 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fee292c-b279-4973-9b39-2a009a781011" path="/var/lib/kubelet/pods/3fee292c-b279-4973-9b39-2a009a781011/volumes" Nov 02 14:30:21.079340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.079325 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" path="/var/lib/kubelet/pods/54b19435-f528-48b4-b807-70e9c6c80abf/volumes" Nov 02 14:30:21.079717 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.079700 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" path="/var/lib/kubelet/pods/6424d0ca-8aa6-4897-a06b-7168cd41016c/volumes" Nov 02 14:30:21.080405 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.080390 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ba8971-616e-43a8-9122-68d990259d85" path="/var/lib/kubelet/pods/b5ba8971-616e-43a8-9122-68d990259d85/volumes" Nov 02 14:30:21.080760 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.080748 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb841a6-f88a-4139-a862-78938304e503" path="/var/lib/kubelet/pods/fbb841a6-f88a-4139-a862-78938304e503/volumes" Nov 02 14:30:21.129389 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.129357 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lnvv\" (UniqueName: \"kubernetes.io/projected/cee0b62b-657a-4897-8e74-191f9ecb42d6-kube-api-access-7lnvv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.129620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.129597 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee0b62b-657a-4897-8e74-191f9ecb42d6-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.378976 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:21.378886 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:30:21.502150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.502118 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" Nov 02 14:30:21.502150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.502130 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff" event={"ID":"ce56cca9-81f2-433f-baf5-a61fb490f1f5","Type":"ContainerDied","Data":"4f89a35cff06e92850a3369300c19702dc8015c23d3ed3bde30ee44880ca2b80"} Nov 02 14:30:21.502737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.502196 4431 scope.go:122] "RemoveContainer" containerID="d75c09eca2edc958758b8251688b3cd1ff672936be923413ad0257dcb60c4ade" Nov 02 14:30:21.503748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.503729 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" Nov 02 14:30:21.503997 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.503732 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m" event={"ID":"ad7316d0-07ef-4664-9345-d00cd333551d","Type":"ContainerDied","Data":"0016a68d2861289f8cc268a32986d6d2bb502586f2bd97ece2901d202b4d0e3a"} Nov 02 14:30:21.505318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.505300 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" Nov 02 14:30:21.505422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.505347 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp" event={"ID":"cee0b62b-657a-4897-8e74-191f9ecb42d6","Type":"ContainerDied","Data":"8939fbe0db84127ea9a4a986513afc7ada7ceaf970c9b43190cd19eb4db63036"} Nov 02 14:30:21.508618 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.508599 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" Nov 02 14:30:21.508714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.508616 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4" event={"ID":"c1aea439-9e6a-42dd-a70f-e30d47a19f19","Type":"ContainerDied","Data":"ac793e8dc184c7d7c782a05f72ab440890ca97ab2d31aa7537608d204a5fba98"} Nov 02 14:30:21.517382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.517354 4431 scope.go:122] "RemoveContainer" containerID="cf719991ecc87683c2d9fbf5781ff52a27ef31c793f0030929b63e8ba986cd10" Nov 02 14:30:21.526321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.526301 4431 scope.go:122] "RemoveContainer" containerID="54a49f5e6b729988f9619b14c241c7478932503e86a4a06d0023d460e66520a7" Nov 02 14:30:21.535241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.535224 4431 scope.go:122] "RemoveContainer" containerID="511b71b231f08f7fd7972a13f9627ef9ef825ab243aa90174d29e38137c92943" Nov 02 14:30:21.548170 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.548151 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4"] Nov 02 14:30:21.572268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.572233 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m"] Nov 02 14:30:21.586901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.586868 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-673ba8-29893830-c2b8m"] Nov 02 14:30:21.590953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.590930 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx"] Nov 02 14:30:21.635749 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.635611 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4"] Nov 02 14:30:21.643429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.643395 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-944506-29893830-75fv4"] Nov 02 14:30:21.682043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.682013 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff"] Nov 02 14:30:21.684161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.684135 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-12b8da-29893830-drmff"] Nov 02 14:30:21.721111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.721002 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp"] Nov 02 14:30:21.726109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.726083 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bb4ef-29893830-8r6jp"] Nov 02 14:30:21.726239 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.726229 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:21.764736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.764709 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c1bd66-084f-49ff-87f5-4111ce798e5b-kube-api-access-fksjz\" (UniqueName: \"kubernetes.io/projected/49c1bd66-084f-49ff-87f5-4111ce798e5b-kube-api-access-fksjz\") pod \"49c1bd66-084f-49ff-87f5-4111ce798e5b\" (UID: \"49c1bd66-084f-49ff-87f5-4111ce798e5b\") " Nov 02 14:30:21.764887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.764761 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c1bd66-084f-49ff-87f5-4111ce798e5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c1bd66-084f-49ff-87f5-4111ce798e5b-maas-api-service-ca\") pod \"49c1bd66-084f-49ff-87f5-4111ce798e5b\" (UID: \"49c1bd66-084f-49ff-87f5-4111ce798e5b\") " Nov 02 14:30:21.765120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.765097 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c1bd66-084f-49ff-87f5-4111ce798e5b-maas-api-service-ca" pod "49c1bd66-084f-49ff-87f5-4111ce798e5b" (UID: "49c1bd66-084f-49ff-87f5-4111ce798e5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:21.765333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.765320 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c1bd66-084f-49ff-87f5-4111ce798e5b-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.767027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.767005 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c1bd66-084f-49ff-87f5-4111ce798e5b-kube-api-access-fksjz" pod "49c1bd66-084f-49ff-87f5-4111ce798e5b" (UID: "49c1bd66-084f-49ff-87f5-4111ce798e5b"). InnerVolumeSpecName "kube-api-access-fksjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:21.769015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.768996 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:21.866318 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.866290 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ef1614-c978-4046-85fe-15c48736d0da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ef1614-c978-4046-85fe-15c48736d0da-maas-api-service-ca\") pod \"76ef1614-c978-4046-85fe-15c48736d0da\" (UID: \"76ef1614-c978-4046-85fe-15c48736d0da\") " Nov 02 14:30:21.866651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.866403 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ef1614-c978-4046-85fe-15c48736d0da-kube-api-access-dxpw8\" (UniqueName: \"kubernetes.io/projected/76ef1614-c978-4046-85fe-15c48736d0da-kube-api-access-dxpw8\") pod \"76ef1614-c978-4046-85fe-15c48736d0da\" (UID: \"76ef1614-c978-4046-85fe-15c48736d0da\") " Nov 02 14:30:21.866832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.866685 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ef1614-c978-4046-85fe-15c48736d0da-maas-api-service-ca" pod "76ef1614-c978-4046-85fe-15c48736d0da" (UID: "76ef1614-c978-4046-85fe-15c48736d0da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:21.867158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.867136 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ef1614-c978-4046-85fe-15c48736d0da-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.867158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.867159 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fksjz\" (UniqueName: \"kubernetes.io/projected/49c1bd66-084f-49ff-87f5-4111ce798e5b-kube-api-access-fksjz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:21.869525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.869495 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ef1614-c978-4046-85fe-15c48736d0da-kube-api-access-dxpw8" pod "76ef1614-c978-4046-85fe-15c48736d0da" (UID: "76ef1614-c978-4046-85fe-15c48736d0da"). InnerVolumeSpecName "kube-api-access-dxpw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:21.968287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:21.968222 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxpw8\" (UniqueName: \"kubernetes.io/projected/76ef1614-c978-4046-85fe-15c48736d0da-kube-api-access-dxpw8\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:22.523644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.523612 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" event={"ID":"49c1bd66-084f-49ff-87f5-4111ce798e5b","Type":"ContainerDied","Data":"b39df9383885bdd79e037426a664b6b9335c99cf8dc4180111ce41c54864db56"} Nov 02 14:30:22.524238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.523673 4431 scope.go:122] "RemoveContainer" containerID="3d8333d1a87a9ec3134616ccdfc0f74258445226029b179191c0008e624166a0" Nov 02 14:30:22.524238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.523622 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4" Nov 02 14:30:22.527212 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.527182 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" Nov 02 14:30:22.527322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.527190 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx" event={"ID":"76ef1614-c978-4046-85fe-15c48736d0da","Type":"ContainerDied","Data":"140e6823b146774cc366da5cedb6b0474a14e112ef90060b0a7a675d5d83d435"} Nov 02 14:30:22.534522 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.534507 4431 scope.go:122] "RemoveContainer" containerID="fb40bd8263a9991a05a01644359636883d5707fa76d6811cfec4c906b6435dd0" Nov 02 14:30:22.640257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.640232 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx"] Nov 02 14:30:22.663415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.663384 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-623b4b-29893830-29bkx"] Nov 02 14:30:22.703075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.703034 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4"] Nov 02 14:30:22.710155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:22.710119 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c0c28f-29893830-fwlj4"] Nov 02 14:30:23.077492 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:23.077461 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" path="/var/lib/kubelet/pods/49c1bd66-084f-49ff-87f5-4111ce798e5b/volumes" Nov 02 14:30:23.077908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:23.077894 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ef1614-c978-4046-85fe-15c48736d0da" path="/var/lib/kubelet/pods/76ef1614-c978-4046-85fe-15c48736d0da/volumes" Nov 02 14:30:23.078271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:23.078259 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" path="/var/lib/kubelet/pods/ad7316d0-07ef-4664-9345-d00cd333551d/volumes" Nov 02 14:30:23.078999 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:23.078979 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" path="/var/lib/kubelet/pods/c1aea439-9e6a-42dd-a70f-e30d47a19f19/volumes" Nov 02 14:30:23.079358 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:23.079346 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" path="/var/lib/kubelet/pods/ce56cca9-81f2-433f-baf5-a61fb490f1f5/volumes" Nov 02 14:30:23.079738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:23.079705 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" path="/var/lib/kubelet/pods/cee0b62b-657a-4897-8e74-191f9ecb42d6/volumes" Nov 02 14:30:25.051380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:25.051347 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7"] Nov 02 14:30:25.063373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:25.063346 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad18a670-zn6w7"] Nov 02 14:30:25.076388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:25.076359 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca69dbf-37f9-4a28-944b-5da842f20991" path="/var/lib/kubelet/pods/5ca69dbf-37f9-4a28-944b-5da842f20991/volumes" Nov 02 14:30:26.819116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.819084 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl"] Nov 02 14:30:26.819952 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.819927 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.819952 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.819951 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.819968 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820011 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820023 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820030 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820046 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820053 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820064 4431 cpu_manager.go:414] "containerMap: removing container" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820070 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820084 4431 cpu_manager.go:414] "containerMap: removing container" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820090 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820104 4431 cpu_manager.go:414] "containerMap: removing container" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820111 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820120 4431 cpu_manager.go:414] "containerMap: removing container" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820128 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820138 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820144 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820156 4431 cpu_manager.go:414] "containerMap: removing container" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820164 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820177 4431 cpu_manager.go:414] "containerMap: removing container" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820183 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820192 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.820188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820201 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820214 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820221 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820231 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820238 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820252 4431 cpu_manager.go:414] "containerMap: removing container" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820259 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820273 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820279 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820300 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820307 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820316 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820322 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820334 4431 cpu_manager.go:414] "containerMap: removing container" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820341 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820354 4431 cpu_manager.go:414] "containerMap: removing container" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820361 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820381 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820387 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820401 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820408 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820419 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820425 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820452 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820458 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820477 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820484 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820621 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820631 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820639 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820647 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820654 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820662 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820671 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820679 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820687 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820696 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820704 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820712 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820720 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820729 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820737 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820745 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820752 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820761 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820768 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c1bd66-084f-49ff-87f5-4111ce798e5b" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820796 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820806 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820815 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820825 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820833 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820842 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7316d0-07ef-4664-9345-d00cd333551d" containerName="cleanup" Nov 02 14:30:26.821343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.820855 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.826571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.826549 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.849093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.849070 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl"] Nov 02 14:30:26.857252 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.857132 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-53f37e-openshift-default-dockercfg-vt8c9\"" Nov 02 14:30:26.926464 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.926434 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb"] Nov 02 14:30:26.926998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.926983 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.926999 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ba8971-616e-43a8-9122-68d990259d85" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927011 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927016 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927029 4431 cpu_manager.go:414] "containerMap: removing container" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927034 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927044 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927048 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927057 4431 cpu_manager.go:414] "containerMap: removing container" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927061 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927073 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.927074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927077 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927088 4431 cpu_manager.go:414] "containerMap: removing container" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927094 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927108 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927112 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fee292c-b279-4973-9b39-2a009a781011" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927196 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce56cca9-81f2-433f-baf5-a61fb490f1f5" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927203 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1aea439-9e6a-42dd-a70f-e30d47a19f19" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927208 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee0b62b-657a-4897-8e74-191f9ecb42d6" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927216 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb841a6-f88a-4139-a862-78938304e503" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927223 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6424d0ca-8aa6-4897-a06b-7168cd41016c" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927229 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b19435-f528-48b4-b807-70e9c6c80abf" containerName="cleanup" Nov 02 14:30:26.927400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.927234 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ef1614-c978-4046-85fe-15c48736d0da" containerName="cleanup" Nov 02 14:30:26.930616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.930599 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:26.947601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.947567 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-data\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.947725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.947674 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-socket\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.947725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.947719 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-token\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.947861 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.947800 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvq25\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-kube-api-access-hvq25\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.947861 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.947842 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-certs\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.948076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.948055 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-envoy\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.948144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.948122 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istiod-ca-cert\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.948196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.948170 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-credential-socket\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.948246 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.948219 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-podinfo\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:26.949247 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.949225 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:30:26.952593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.952568 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:30:26.957094 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:26.957066 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb"] Nov 02 14:30:27.063800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.063598 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-credential-socket\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.063800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.063676 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-podinfo\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.063800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.063739 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfknl\" (UniqueName: \"kubernetes.io/projected/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-kube-api-access-kfknl\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.064137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.063863 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-data\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.064137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.063950 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.064137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.063994 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-socket\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.064137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.064021 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.064137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.064059 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-token\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.064406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.064140 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvq25\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-kube-api-access-hvq25\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.064406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.064179 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-certs\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.064406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.064214 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-envoy\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.064406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.064267 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istiod-ca-cert\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.076840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.065091 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istiod-ca-cert\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.076840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.065393 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-credential-socket\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.076840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.066103 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-data\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.076840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.066350 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-socket\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.076840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.067091 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-certs\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.083800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.079843 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-podinfo\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.084034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.083317 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-envoy\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.113800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.111050 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvq25\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-kube-api-access-hvq25\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.113800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.113260 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-token\") pod \"e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.146025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.145997 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:27.165617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.165585 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfknl\" (UniqueName: \"kubernetes.io/projected/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-kube-api-access-kfknl\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.165748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.165734 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.165850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.165772 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.166494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.166467 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.168400 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.168377 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.190988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.190961 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfknl\" (UniqueName: \"kubernetes.io/projected/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-kube-api-access-kfknl\") pod \"maas-api-revoke-sub-489a8abe-cjbvb\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.247505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.247472 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:27.471863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.471838 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl"] Nov 02 14:30:27.472239 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:27.472213 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ea66e0_2869_4d24_ad6e_f5a47b266f01.slice/crio-ef3a131cf1e9fee13d7ad3741d14b00e69981566271b8a4615fbab0af1b590d7 WatchSource:0}: Error finding container ef3a131cf1e9fee13d7ad3741d14b00e69981566271b8a4615fbab0af1b590d7: Status 404 returned error can't find the container with id ef3a131cf1e9fee13d7ad3741d14b00e69981566271b8a4615fbab0af1b590d7 Nov 02 14:30:27.474865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.474831 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:27.474962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.474898 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:27.474962 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.474928 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:30:27.564557 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:27.564519 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f2609e_f1ba_4cce_9b7e_afbbf672473c.slice/crio-c1e983b209753a192dedefd728ca6cc0ec0314c68abe61d1f96361f8ae8c53f2 WatchSource:0}: Error finding container c1e983b209753a192dedefd728ca6cc0ec0314c68abe61d1f96361f8ae8c53f2: Status 404 returned error can't find the container with id c1e983b209753a192dedefd728ca6cc0ec0314c68abe61d1f96361f8ae8c53f2 Nov 02 14:30:27.564557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.564544 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb"] Nov 02 14:30:27.568401 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.568375 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" event={"ID":"54ea66e0-2869-4d24-ad6e-f5a47b266f01","Type":"ContainerStarted","Data":"f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa"} Nov 02 14:30:27.568508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.568426 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" event={"ID":"54ea66e0-2869-4d24-ad6e-f5a47b266f01","Type":"ContainerStarted","Data":"ef3a131cf1e9fee13d7ad3741d14b00e69981566271b8a4615fbab0af1b590d7"} Nov 02 14:30:27.627433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:27.627369 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" podStartSLOduration=1.6273475419999999 podStartE2EDuration="1.627347542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:27.626222673 +0000 UTC m=+1992.663327608" watchObservedRunningTime="2026-11-02 14:30:27.627347542 +0000 UTC m=+1992.664452476" Nov 02 14:30:28.146360 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:28.146332 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:28.152929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:28.152897 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:28.576966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:28.576931 4431 generic.go:356] "Generic (PLEG): container finished" podID="b5f2609e-f1ba-4cce-9b7e-afbbf672473c" containerID="dc787fb315bc65b0f6f98b00ac2ea8f0ff0270b8e299695e0eed593194e959a5" exitCode=0 Nov 02 14:30:28.577155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:28.577007 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" event={"ID":"b5f2609e-f1ba-4cce-9b7e-afbbf672473c","Type":"ContainerDied","Data":"dc787fb315bc65b0f6f98b00ac2ea8f0ff0270b8e299695e0eed593194e959a5"} Nov 02 14:30:28.577155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:28.577065 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" event={"ID":"b5f2609e-f1ba-4cce-9b7e-afbbf672473c","Type":"ContainerStarted","Data":"c1e983b209753a192dedefd728ca6cc0ec0314c68abe61d1f96361f8ae8c53f2"} Nov 02 14:30:28.577487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:28.577467 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:28.578323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:28.578305 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:29.742289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.742269 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:29.884449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.884372 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25"] Nov 02 14:30:29.885030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.885015 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b5f2609e-f1ba-4cce-9b7e-afbbf672473c" containerName="revoke-keys" Nov 02 14:30:29.885076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.885031 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f2609e-f1ba-4cce-9b7e-afbbf672473c" containerName="revoke-keys" Nov 02 14:30:29.885126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.885117 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f2609e-f1ba-4cce-9b7e-afbbf672473c" containerName="revoke-keys" Nov 02 14:30:29.888494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.888477 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:29.900817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.900795 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-openshift-ca\") pod \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " Nov 02 14:30:29.900919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.900855 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-kube-api-access-kfknl\" (UniqueName: \"kubernetes.io/projected/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-kube-api-access-kfknl\") pod \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " Nov 02 14:30:29.900963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.900941 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-xks-ca\") pod \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\" (UID: \"b5f2609e-f1ba-4cce-9b7e-afbbf672473c\") " Nov 02 14:30:29.901230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.901203 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-openshift-ca" pod "b5f2609e-f1ba-4cce-9b7e-afbbf672473c" (UID: "b5f2609e-f1ba-4cce-9b7e-afbbf672473c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:29.901590 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.901570 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:29.903555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.903531 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-xks-ca" pod "b5f2609e-f1ba-4cce-9b7e-afbbf672473c" (UID: "b5f2609e-f1ba-4cce-9b7e-afbbf672473c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:30:29.903662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.903640 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-kube-api-access-kfknl" pod "b5f2609e-f1ba-4cce-9b7e-afbbf672473c" (UID: "b5f2609e-f1ba-4cce-9b7e-afbbf672473c"). InnerVolumeSpecName "kube-api-access-kfknl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:29.917520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.917495 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-53f37e\"/\"openshift-service-ca.crt\"" Nov 02 14:30:29.917650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.917636 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-53f37e\"/\"kube-root-ca.crt\"" Nov 02 14:30:29.917944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.917924 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25"] Nov 02 14:30:29.918259 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.918241 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-53f37e\"/\"default-dockercfg-jsmhh\"" Nov 02 14:30:29.918326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:29.918317 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-53f37e\"/\"noten-model-53f37e-kserve-self-signed-certs\"" Nov 02 14:30:30.002556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.002520 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-dshm\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.002724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.002664 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-model-cache\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.002724 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.002711 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwt6\" (UniqueName: \"kubernetes.io/projected/d62a6204-4240-4e7b-9162-43da6d5da680-kube-api-access-fpwt6\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.002886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.002733 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-tmp-dir\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.002886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.002814 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-home\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.003000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.002920 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6204-4240-4e7b-9162-43da6d5da680-tls-certs\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.003117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.003100 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:30.003182 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.003122 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfknl\" (UniqueName: \"kubernetes.io/projected/b5f2609e-f1ba-4cce-9b7e-afbbf672473c-kube-api-access-kfknl\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:30.104421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.104372 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-model-cache\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.104631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.104464 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwt6\" (UniqueName: \"kubernetes.io/projected/d62a6204-4240-4e7b-9162-43da6d5da680-kube-api-access-fpwt6\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.104631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.104496 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-tmp-dir\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.104631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.104533 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-home\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.104631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.104567 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6204-4240-4e7b-9162-43da6d5da680-tls-certs\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.104895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.104718 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-dshm\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.115845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.106182 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-model-cache\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.115845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.106430 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-home\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.115845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.107468 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-tmp-dir\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.115845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.111082 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-dshm\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.136895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.130175 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6204-4240-4e7b-9162-43da6d5da680-tls-certs\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.156801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.144233 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwt6\" (UniqueName: \"kubernetes.io/projected/d62a6204-4240-4e7b-9162-43da6d5da680-kube-api-access-fpwt6\") pod \"noten-model-53f37e-kserve-ffc55cc-sdj25\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.201071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.201037 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:30.521902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.521865 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25"] Nov 02 14:30:30.524416 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:30.524386 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd62a6204_4240_4e7b_9162_43da6d5da680.slice/crio-230641552b647c103a77ae265da70ba0a17d05e7cbe858a009f9bed50e76bef6 WatchSource:0}: Error finding container 230641552b647c103a77ae265da70ba0a17d05e7cbe858a009f9bed50e76bef6: Status 404 returned error can't find the container with id 230641552b647c103a77ae265da70ba0a17d05e7cbe858a009f9bed50e76bef6 Nov 02 14:30:30.611840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.611809 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" event={"ID":"d62a6204-4240-4e7b-9162-43da6d5da680","Type":"ContainerStarted","Data":"230641552b647c103a77ae265da70ba0a17d05e7cbe858a009f9bed50e76bef6"} Nov 02 14:30:30.613638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.613611 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" event={"ID":"b5f2609e-f1ba-4cce-9b7e-afbbf672473c","Type":"ContainerDied","Data":"c1e983b209753a192dedefd728ca6cc0ec0314c68abe61d1f96361f8ae8c53f2"} Nov 02 14:30:30.614116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.613655 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb" Nov 02 14:30:30.614116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:30.613659 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1e983b209753a192dedefd728ca6cc0ec0314c68abe61d1f96361f8ae8c53f2" Nov 02 14:30:31.623321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:31.623284 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" event={"ID":"d62a6204-4240-4e7b-9162-43da6d5da680","Type":"ContainerStarted","Data":"2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7"} Nov 02 14:30:31.671141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:31.671098 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" podStartSLOduration=2.6710834329999997 podStartE2EDuration="2.671083433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:31.667206009 +0000 UTC m=+1996.704310941" watchObservedRunningTime="2026-11-02 14:30:31.671083433 +0000 UTC m=+1996.708188364" Nov 02 14:30:32.201381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:32.201345 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:40.201642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:40.201599 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:40.215216 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:40.215192 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:40.695604 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:40.695575 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:30:48.820633 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:48.820602 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5"] Nov 02 14:30:48.825338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:48.825320 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:48.847469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:48.847440 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5"] Nov 02 14:30:48.855159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:48.855136 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:30:48.855259 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:48.855162 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:30:48.947433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:48.947404 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:48.947627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:48.947570 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:48.947627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:48.947612 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ng2z\" (UniqueName: \"kubernetes.io/projected/61d644dc-5b60-4624-8fdb-7a05b09504b1-kube-api-access-8ng2z\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:49.051800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.049761 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:49.051800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.049862 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ng2z\" (UniqueName: \"kubernetes.io/projected/61d644dc-5b60-4624-8fdb-7a05b09504b1-kube-api-access-8ng2z\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:49.051800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.050356 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:49.051800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.050595 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:49.059802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.057938 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25"] Nov 02 14:30:49.073296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.073225 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:49.076351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.076325 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ng2z\" (UniqueName: \"kubernetes.io/projected/61d644dc-5b60-4624-8fdb-7a05b09504b1-kube-api-access-8ng2z\") pod \"maas-api-revoke-sub-2b1bd63e-vzdc5\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:49.144060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.144028 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:49.301799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.293649 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl"] Nov 02 14:30:49.301799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.294131 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" podUID="54ea66e0-2869-4d24-ad6e-f5a47b266f01" containerName="istio-proxy" containerID="cri-o://f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa" gracePeriod=30 Nov 02 14:30:49.752958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:49.752928 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5"] Nov 02 14:30:49.754597 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:30:49.754567 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d644dc_5b60_4624_8fdb_7a05b09504b1.slice/crio-176d3dcc151e71e091b016423be5313a4d0d11f498b1fae933beeb0bf61f7b82 WatchSource:0}: Error finding container 176d3dcc151e71e091b016423be5313a4d0d11f498b1fae933beeb0bf61f7b82: Status 404 returned error can't find the container with id 176d3dcc151e71e091b016423be5313a4d0d11f498b1fae933beeb0bf61f7b82 Nov 02 14:30:50.760271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:50.760237 4431 generic.go:356] "Generic (PLEG): container finished" podID="61d644dc-5b60-4624-8fdb-7a05b09504b1" containerID="08b457406a97ff27851ba41748184bfb918cfb56f2d25f06b5f35818a2846b4d" exitCode=0 Nov 02 14:30:50.760634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:50.760288 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" event={"ID":"61d644dc-5b60-4624-8fdb-7a05b09504b1","Type":"ContainerDied","Data":"08b457406a97ff27851ba41748184bfb918cfb56f2d25f06b5f35818a2846b4d"} Nov 02 14:30:50.760634 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:50.760343 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" event={"ID":"61d644dc-5b60-4624-8fdb-7a05b09504b1","Type":"ContainerStarted","Data":"176d3dcc151e71e091b016423be5313a4d0d11f498b1fae933beeb0bf61f7b82"} Nov 02 14:30:51.283094 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:51.283060 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:30:51.923195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:51.923165 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:52.010141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.010107 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d644dc-5b60-4624-8fdb-7a05b09504b1-kube-api-access-8ng2z\" (UniqueName: \"kubernetes.io/projected/61d644dc-5b60-4624-8fdb-7a05b09504b1-kube-api-access-8ng2z\") pod \"61d644dc-5b60-4624-8fdb-7a05b09504b1\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " Nov 02 14:30:52.010343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.010310 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-openshift-ca\") pod \"61d644dc-5b60-4624-8fdb-7a05b09504b1\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " Nov 02 14:30:52.010452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.010403 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-xks-ca\") pod \"61d644dc-5b60-4624-8fdb-7a05b09504b1\" (UID: \"61d644dc-5b60-4624-8fdb-7a05b09504b1\") " Nov 02 14:30:52.010732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.010702 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-openshift-ca" pod "61d644dc-5b60-4624-8fdb-7a05b09504b1" (UID: "61d644dc-5b60-4624-8fdb-7a05b09504b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:52.011006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.010983 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:52.012451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.012428 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d644dc-5b60-4624-8fdb-7a05b09504b1-kube-api-access-8ng2z" pod "61d644dc-5b60-4624-8fdb-7a05b09504b1" (UID: "61d644dc-5b60-4624-8fdb-7a05b09504b1"). InnerVolumeSpecName "kube-api-access-8ng2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:52.012721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.012695 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-xks-ca" pod "61d644dc-5b60-4624-8fdb-7a05b09504b1" (UID: "61d644dc-5b60-4624-8fdb-7a05b09504b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:30:52.111966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.111937 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d644dc-5b60-4624-8fdb-7a05b09504b1-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:52.111966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.111963 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ng2z\" (UniqueName: \"kubernetes.io/projected/61d644dc-5b60-4624-8fdb-7a05b09504b1-kube-api-access-8ng2z\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:52.775557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.775526 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" Nov 02 14:30:52.775557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.775552 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5" event={"ID":"61d644dc-5b60-4624-8fdb-7a05b09504b1","Type":"ContainerDied","Data":"176d3dcc151e71e091b016423be5313a4d0d11f498b1fae933beeb0bf61f7b82"} Nov 02 14:30:52.775759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:52.775597 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="176d3dcc151e71e091b016423be5313a4d0d11f498b1fae933beeb0bf61f7b82" Nov 02 14:30:54.493056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.493034 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:54.641879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.641853 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-kube-api-access-hvq25\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-kube-api-access-hvq25\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.641898 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-token\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-token\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.641927 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istiod-ca-cert\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.641963 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-data\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642014 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-podinfo\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642067 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-envoy\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642146 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-credential-socket\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642205 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-certs\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642274 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642235 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-socket\") pod \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\" (UID: \"54ea66e0-2869-4d24-ad6e-f5a47b266f01\") " Nov 02 14:30:54.642520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642282 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-data" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:54.642520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642397 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istiod-ca-cert" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:30:54.642642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642571 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-socket" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:54.642705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642684 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-certs" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:54.642849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642823 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-credential-socket" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:54.642945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642896 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.642945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642914 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.642945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642927 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.642945 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.642938 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.644173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.644144 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-kube-api-access-hvq25" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "kube-api-access-hvq25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:54.644358 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.644340 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-token" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:30:54.644731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.644714 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-envoy" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:30:54.644834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.644818 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-podinfo" pod "54ea66e0-2869-4d24-ad6e-f5a47b266f01" (UID: "54ea66e0-2869-4d24-ad6e-f5a47b266f01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:30:54.744449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.744421 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.744449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.744446 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.744609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.744456 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ea66e0-2869-4d24-ad6e-f5a47b266f01-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.744609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.744464 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvq25\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-kube-api-access-hvq25\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.744609 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.744475 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ea66e0-2869-4d24-ad6e-f5a47b266f01-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:30:54.791754 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.791722 4431 generic.go:356] "Generic (PLEG): container finished" podID="54ea66e0-2869-4d24-ad6e-f5a47b266f01" containerID="f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa" exitCode=0 Nov 02 14:30:54.791884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.791790 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" Nov 02 14:30:54.791884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.791809 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" event={"ID":"54ea66e0-2869-4d24-ad6e-f5a47b266f01","Type":"ContainerDied","Data":"f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa"} Nov 02 14:30:54.791884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.791875 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl" event={"ID":"54ea66e0-2869-4d24-ad6e-f5a47b266f01","Type":"ContainerDied","Data":"ef3a131cf1e9fee13d7ad3741d14b00e69981566271b8a4615fbab0af1b590d7"} Nov 02 14:30:54.792009 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.791927 4431 scope.go:122] "RemoveContainer" containerID="f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa" Nov 02 14:30:54.803432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.803415 4431 scope.go:122] "RemoveContainer" containerID="f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa" Nov 02 14:30:54.803695 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:30:54.803671 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa\": container with ID starting with f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa not found: ID does not exist" containerID="f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa" Nov 02 14:30:54.803823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.803706 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa"} err="failed to get container status \"f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa\": rpc error: code = NotFound desc = could not find container \"f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa\": container with ID starting with f6092ef1f6dfc5134b3151ae497c7f8421a90d9e16d48b85e8315b031061c1aa not found: ID does not exist" Nov 02 14:30:54.852922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.852898 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl"] Nov 02 14:30:54.858159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:54.858138 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-53f37e-openshift-default-6766bfb7f-qrkfl"] Nov 02 14:30:55.076016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:55.075945 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ea66e0-2869-4d24-ad6e-f5a47b266f01" path="/var/lib/kubelet/pods/54ea66e0-2869-4d24-ad6e-f5a47b266f01/volumes" Nov 02 14:30:57.650823 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:30:57.650764 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:31:00.167514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:00.167470 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:00.266800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:00.266318 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:00.282799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:00.281889 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:00.419460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:00.419353 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:00.593270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:00.593220 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" podUID="537d9533-ce0c-4f93-be2f-684bce472885" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:01.095810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.094892 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:31:01.095810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.094961 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:01.138801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.137860 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:31:01.138801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.137920 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:01.385820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.385703 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:01.465227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.423918 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:01.488380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.488345 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:31:01.535919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.535876 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:01.635801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.634922 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" podUID="537d9533-ce0c-4f93-be2f-684bce472885" containerName="main" probeResult="failure" output="Get \"https://10.133.0.197:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 14:31:01.635801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.635647 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:31:01.764814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.764611 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:31:01.765889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:01.765867 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:31:02.066112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.066028 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb"] Nov 02 14:31:02.067261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.067236 4431 cpu_manager.go:414] "containerMap: removing container" podUID="54ea66e0-2869-4d24-ad6e-f5a47b266f01" containerName="istio-proxy" Nov 02 14:31:02.067487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.067471 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ea66e0-2869-4d24-ad6e-f5a47b266f01" containerName="istio-proxy" Nov 02 14:31:02.067624 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.067611 4431 cpu_manager.go:414] "containerMap: removing container" podUID="61d644dc-5b60-4624-8fdb-7a05b09504b1" containerName="revoke-keys" Nov 02 14:31:02.067723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.067711 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d644dc-5b60-4624-8fdb-7a05b09504b1" containerName="revoke-keys" Nov 02 14:31:02.068015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.067997 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ea66e0-2869-4d24-ad6e-f5a47b266f01" containerName="istio-proxy" Nov 02 14:31:02.068147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.068134 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d644dc-5b60-4624-8fdb-7a05b09504b1" containerName="revoke-keys" Nov 02 14:31:02.081850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.081827 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.108015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.107987 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb"] Nov 02 14:31:02.117992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.117967 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e07583-openshift-default-dockercfg-cwg6k\"" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.248638 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istiod-ca-cert\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.248698 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-socket\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.248746 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-data\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.248810 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-envoy\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.248945 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ztz\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-kube-api-access-t8ztz\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.248983 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-credential-socket\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.249019 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-token\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.249079 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-podinfo\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.259799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.249191 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-certs\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.357141 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ztz\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-kube-api-access-t8ztz\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.357610 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-credential-socket\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.357822 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-token\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.358052 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-podinfo\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.358517 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-certs\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.358766 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istiod-ca-cert\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.358987 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-socket\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.359230 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-data\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.359479 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-envoy\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.359844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.359756 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-credential-socket\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.360927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.360849 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-socket\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.361059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.360972 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-certs\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.361545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.361519 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-data\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.362519 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.362392 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istiod-ca-cert\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.377652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.377616 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-podinfo\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.391701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.391646 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ztz\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-kube-api-access-t8ztz\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.399229 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.399190 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-envoy\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.404572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.404542 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-token\") pod \"e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.429065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.429027 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:02.807260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.806807 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb"] Nov 02 14:31:02.809648 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:31:02.809616 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb08832_f1bd_44e1_bf69_b9d3a298980b.slice/crio-7669f44a8cc3a3afc55f78c62953a70bd221972ac91693fc261edd6f4ad59ded WatchSource:0}: Error finding container 7669f44a8cc3a3afc55f78c62953a70bd221972ac91693fc261edd6f4ad59ded: Status 404 returned error can't find the container with id 7669f44a8cc3a3afc55f78c62953a70bd221972ac91693fc261edd6f4ad59ded Nov 02 14:31:02.813057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.813017 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:02.813158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.813098 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:02.813158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.813135 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:02.944134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.944097 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" event={"ID":"3eb08832-f1bd-44e1-bf69-b9d3a298980b","Type":"ContainerStarted","Data":"a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c"} Nov 02 14:31:02.944291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:02.944159 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" event={"ID":"3eb08832-f1bd-44e1-bf69-b9d3a298980b","Type":"ContainerStarted","Data":"7669f44a8cc3a3afc55f78c62953a70bd221972ac91693fc261edd6f4ad59ded"} Nov 02 14:31:03.012535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:03.012446 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" podStartSLOduration=1.012427758 podStartE2EDuration="1.012427758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:03.011852078 +0000 UTC m=+2028.048957012" watchObservedRunningTime="2026-11-02 14:31:03.012427758 +0000 UTC m=+2028.049532694" Nov 02 14:31:03.429566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:03.429526 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:03.431543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:03.431515 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 14:31:03.431695 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:03.431563 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 14:31:04.096643 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.096591 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" podUID="d62a6204-4240-4e7b-9162-43da6d5da680" containerName="main" containerID="cri-o://2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7" gracePeriod=45 Nov 02 14:31:04.427881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.427858 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:31:04.434149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.434119 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:04.607967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.607926 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-dshm\") pod \"d62a6204-4240-4e7b-9162-43da6d5da680\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " Nov 02 14:31:04.608178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.607981 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-tmp-dir\") pod \"d62a6204-4240-4e7b-9162-43da6d5da680\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " Nov 02 14:31:04.608178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608015 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62a6204-4240-4e7b-9162-43da6d5da680-tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6204-4240-4e7b-9162-43da6d5da680-tls-certs\") pod \"d62a6204-4240-4e7b-9162-43da6d5da680\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " Nov 02 14:31:04.608178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608086 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62a6204-4240-4e7b-9162-43da6d5da680-kube-api-access-fpwt6\" (UniqueName: \"kubernetes.io/projected/d62a6204-4240-4e7b-9162-43da6d5da680-kube-api-access-fpwt6\") pod \"d62a6204-4240-4e7b-9162-43da6d5da680\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " Nov 02 14:31:04.608356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608189 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-home\") pod \"d62a6204-4240-4e7b-9162-43da6d5da680\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " Nov 02 14:31:04.608356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608284 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-tmp-dir" pod "d62a6204-4240-4e7b-9162-43da6d5da680" (UID: "d62a6204-4240-4e7b-9162-43da6d5da680"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:04.608356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608347 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-model-cache\") pod \"d62a6204-4240-4e7b-9162-43da6d5da680\" (UID: \"d62a6204-4240-4e7b-9162-43da6d5da680\") " Nov 02 14:31:04.608527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608459 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-home" pod "d62a6204-4240-4e7b-9162-43da6d5da680" (UID: "d62a6204-4240-4e7b-9162-43da6d5da680"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:04.608755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608721 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-model-cache" pod "d62a6204-4240-4e7b-9162-43da6d5da680" (UID: "d62a6204-4240-4e7b-9162-43da6d5da680"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:04.608968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608944 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:04.608968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608969 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:04.609169 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.608982 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:04.610832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.610805 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62a6204-4240-4e7b-9162-43da6d5da680-tls-certs" pod "d62a6204-4240-4e7b-9162-43da6d5da680" (UID: "d62a6204-4240-4e7b-9162-43da6d5da680"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:31:04.610928 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.610907 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62a6204-4240-4e7b-9162-43da6d5da680-kube-api-access-fpwt6" pod "d62a6204-4240-4e7b-9162-43da6d5da680" (UID: "d62a6204-4240-4e7b-9162-43da6d5da680"). InnerVolumeSpecName "kube-api-access-fpwt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:04.610928 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.610906 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-dshm" pod "d62a6204-4240-4e7b-9162-43da6d5da680" (UID: "d62a6204-4240-4e7b-9162-43da6d5da680"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:04.710388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.710322 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62a6204-4240-4e7b-9162-43da6d5da680-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:04.710388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.710348 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62a6204-4240-4e7b-9162-43da6d5da680-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:04.710388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.710359 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpwt6\" (UniqueName: \"kubernetes.io/projected/d62a6204-4240-4e7b-9162-43da6d5da680-kube-api-access-fpwt6\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:04.959868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.959834 4431 generic.go:356] "Generic (PLEG): container finished" podID="d62a6204-4240-4e7b-9162-43da6d5da680" containerID="2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7" exitCode=0 Nov 02 14:31:04.960070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.959894 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" Nov 02 14:31:04.960070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.959925 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" event={"ID":"d62a6204-4240-4e7b-9162-43da6d5da680","Type":"ContainerDied","Data":"2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7"} Nov 02 14:31:04.960070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.959975 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25" event={"ID":"d62a6204-4240-4e7b-9162-43da6d5da680","Type":"ContainerDied","Data":"230641552b647c103a77ae265da70ba0a17d05e7cbe858a009f9bed50e76bef6"} Nov 02 14:31:04.960070 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.960005 4431 scope.go:122] "RemoveContainer" containerID="2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7" Nov 02 14:31:04.960368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.960355 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:04.961613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.961591 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:31:04.997718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.997700 4431 scope.go:122] "RemoveContainer" containerID="2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7" Nov 02 14:31:04.998109 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:31:04.998089 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7\": container with ID starting with 2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7 not found: ID does not exist" containerID="2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7" Nov 02 14:31:04.998183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:04.998119 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7"} err="failed to get container status \"2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7\": rpc error: code = NotFound desc = could not find container \"2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7\": container with ID starting with 2db284d30b9139285347c48d7d216965ae324eeba425a4479b0b2d793ad1fec7 not found: ID does not exist" Nov 02 14:31:05.109403 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:05.109371 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25"] Nov 02 14:31:05.112267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:05.112240 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-53f37e/noten-model-53f37e-kserve-ffc55cc-sdj25"] Nov 02 14:31:07.078904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:07.078868 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62a6204-4240-4e7b-9162-43da6d5da680" path="/var/lib/kubelet/pods/d62a6204-4240-4e7b-9162-43da6d5da680/volumes" Nov 02 14:31:10.003600 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:10.003566 4431 generic.go:356] "Generic (PLEG): container finished" podID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerID="ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692" exitCode=0 Nov 02 14:31:10.004085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:10.003636 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" event={"ID":"c04e0282-f36c-44cf-8a20-af39dcb9302c","Type":"ContainerDied","Data":"ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692"} Nov 02 14:31:10.004085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:10.004023 4431 scope.go:122] "RemoveContainer" containerID="ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692" Nov 02 14:31:10.005498 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:10.005468 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:10.005574 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:10.005539 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:10.005621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:10.005574 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:11.014004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:11.013970 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" event={"ID":"c04e0282-f36c-44cf-8a20-af39dcb9302c","Type":"ContainerStarted","Data":"7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04"} Nov 02 14:31:11.374003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:11.373862 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:31:11.658523 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:11.658492 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:31:11.659610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:11.659587 4431 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 14:31:11.659740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:11.659632 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 14:31:12.657944 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:12.657903 4431 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 14:31:12.658375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:12.657986 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 14:31:13.658413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:13.658378 4431 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 14:31:13.658875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:13.658426 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 14:31:14.658681 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:14.658648 4431 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 14:31:14.659099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:14.658696 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 14:31:15.668810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:15.661421 4431 patch_prober.go:29] interesting pod/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Nov 02 14:31:15.679973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:15.679902 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Nov 02 14:31:16.673713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:16.672152 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:31:16.685104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:16.685071 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:31:17.695794 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:17.695749 4431 scope.go:122] "RemoveContainer" containerID="b1e3652e6832bde217ff33e838a4cb3972a0576e8703a1e7e5b80af37e1325f6" Nov 02 14:31:21.306837 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:31:21.306767 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:31:26.168376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:26.168341 4431 generic.go:356] "Generic (PLEG): container finished" podID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerID="1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376" exitCode=0 Nov 02 14:31:26.169244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:26.169217 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" event={"ID":"61d580e7-f40b-47ea-b707-e1c6c3e144ac","Type":"ContainerDied","Data":"1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376"} Nov 02 14:31:26.169865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:26.169846 4431 scope.go:122] "RemoveContainer" containerID="1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376" Nov 02 14:31:26.172973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:26.172933 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:26.173071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:26.173029 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:26.173114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:26.173074 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:31:27.198806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:27.196879 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" event={"ID":"61d580e7-f40b-47ea-b707-e1c6c3e144ac","Type":"ContainerStarted","Data":"a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b"} Nov 02 14:31:27.520531 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:27.520434 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:31:27.520736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:27.520707 4431 patch_prober.go:29] interesting pod/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 14:31:27.520858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:27.520795 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 14:31:28.453900 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.453863 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds"] Nov 02 14:31:28.454467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.454450 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d62a6204-4240-4e7b-9162-43da6d5da680" containerName="main" Nov 02 14:31:28.454467 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.454465 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62a6204-4240-4e7b-9162-43da6d5da680" containerName="main" Nov 02 14:31:28.454639 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.454569 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62a6204-4240-4e7b-9162-43da6d5da680" containerName="main" Nov 02 14:31:28.458306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.458284 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.502771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.498061 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-778b2bce\"/\"kube-root-ca.crt\"" Nov 02 14:31:28.502771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.498179 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-778b2bce\"/\"default-dockercfg-bjpqd\"" Nov 02 14:31:28.502771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.498239 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-778b2bce\"/\"e2e-lifecycle-model-778b2bce-kserve-self-signed-certs\"" Nov 02 14:31:28.502771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.498706 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-778b2bce\"/\"openshift-service-ca.crt\"" Nov 02 14:31:28.507744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.507711 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds"] Nov 02 14:31:28.521026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.520984 4431 patch_prober.go:29] interesting pod/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 14:31:28.521309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.521274 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 14:31:28.552693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.552663 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-home\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.552858 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.552709 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-model-cache\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.552931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.552889 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-tmp-dir\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.552992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.552971 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-dshm\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.553052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.553020 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09604db3-ce99-474f-9448-d937b4461323-tls-certs\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.553145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.553128 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/09604db3-ce99-474f-9448-d937b4461323-kube-api-access-jtwjz\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.654597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.654560 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/09604db3-ce99-474f-9448-d937b4461323-kube-api-access-jtwjz\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.654770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.654689 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-home\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.654770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.654711 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-model-cache\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.654770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.654744 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-tmp-dir\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.654770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.654766 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-dshm\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.655004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.654977 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09604db3-ce99-474f-9448-d937b4461323-tls-certs\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.655145 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.655128 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-model-cache\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.655243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.655218 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-tmp-dir\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.655556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.655533 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-home\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.657294 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.657268 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-dshm\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.657681 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.657658 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09604db3-ce99-474f-9448-d937b4461323-tls-certs\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.682798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.680912 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/09604db3-ce99-474f-9448-d937b4461323-kube-api-access-jtwjz\") pod \"e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:28.808852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:28.808085 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:29.192801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:29.189588 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds"] Nov 02 14:31:29.216912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:29.216868 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" event={"ID":"09604db3-ce99-474f-9448-d937b4461323","Type":"ContainerStarted","Data":"ced99a589cb77af97857ff321a1be978f4d0f31e549fa8164d2f989295bfe55d"} Nov 02 14:31:29.521487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:29.521408 4431 patch_prober.go:29] interesting pod/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 14:31:29.521487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:29.521461 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 14:31:30.226497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:30.226457 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" event={"ID":"09604db3-ce99-474f-9448-d937b4461323","Type":"ContainerStarted","Data":"04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2"} Nov 02 14:31:30.292800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:30.290234 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" podStartSLOduration=2.290217137 podStartE2EDuration="2.290217137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:30.286204983 +0000 UTC m=+2055.323309916" watchObservedRunningTime="2026-11-02 14:31:30.290217137 +0000 UTC m=+2055.327322073" Nov 02 14:31:30.520879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:30.520772 4431 patch_prober.go:29] interesting pod/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 14:31:30.520879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:30.520857 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 14:31:30.808907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:30.808820 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:31.521086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:31.521046 4431 patch_prober.go:29] interesting pod/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 14:31:31.521280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:31.521116 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 14:31:32.428473 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:32.428441 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:31:32.526295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:32.526269 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:31:33.249213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:33.249184 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:31:37.623039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:37.623008 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:31:38.809425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:38.809384 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:38.827737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:38.827703 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:39.325205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:39.325172 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:31:39.539626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:39.539594 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn"] Nov 02 14:31:39.539891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:39.539861 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" containerID="cri-o://1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c" gracePeriod=30 Nov 02 14:31:44.820244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.820186 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:31:44.936188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936155 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-envoy\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936252 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-token\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-token\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936288 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-socket\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936347 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-podinfo\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936374 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-credential-socket\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936509 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-certs\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936608 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-kube-api-access-8mc8c\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-kube-api-access-8mc8c\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936693 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-socket" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:44.936736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936686 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-credential-socket" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:44.936736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936706 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istiod-ca-cert\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936853 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-data\") pod \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\" (UID: \"fe2e3387-dee2-46b2-ad82-fcd52485a3aa\") " Nov 02 14:31:44.936958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.936868 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-certs" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:44.937079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.937033 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istiod-ca-cert" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:31:44.937172 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.937152 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-data" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:44.937581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.937555 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:44.937581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.937582 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:44.937803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.937595 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:44.937803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.937610 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:44.937803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.937624 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:44.938538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.938509 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-envoy" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:31:44.938855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.938835 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-podinfo" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:31:44.938989 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.938971 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-token" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:44.939051 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:44.939002 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-kube-api-access-8mc8c" pod "fe2e3387-dee2-46b2-ad82-fcd52485a3aa" (UID: "fe2e3387-dee2-46b2-ad82-fcd52485a3aa"). InnerVolumeSpecName "kube-api-access-8mc8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:31:45.038348 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.038325 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mc8c\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-kube-api-access-8mc8c\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:45.038348 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.038347 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:45.038486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.038356 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:45.038486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.038363 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2e3387-dee2-46b2-ad82-fcd52485a3aa-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:31:45.357205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.357122 4431 generic.go:356] "Generic (PLEG): container finished" podID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerID="1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c" exitCode=0 Nov 02 14:31:45.357205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.357161 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" event={"ID":"fe2e3387-dee2-46b2-ad82-fcd52485a3aa","Type":"ContainerDied","Data":"1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c"} Nov 02 14:31:45.357205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.357203 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" event={"ID":"fe2e3387-dee2-46b2-ad82-fcd52485a3aa","Type":"ContainerDied","Data":"8f196a995db3b7ec7fa52af8373afff4fbedc0204aacf66c7245defb2f47dadd"} Nov 02 14:31:45.357454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.357225 4431 scope.go:122] "RemoveContainer" containerID="1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c" Nov 02 14:31:45.357454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.357163 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn" Nov 02 14:31:45.367955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.367938 4431 scope.go:122] "RemoveContainer" containerID="2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70" Nov 02 14:31:45.377276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.377259 4431 scope.go:122] "RemoveContainer" containerID="1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c" Nov 02 14:31:45.377569 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:31:45.377549 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c\": container with ID starting with 1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c not found: ID does not exist" containerID="1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c" Nov 02 14:31:45.377682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.377581 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c"} err="failed to get container status \"1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c\": rpc error: code = NotFound desc = could not find container \"1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c\": container with ID starting with 1d86229ed6c0fa877d0bba9d06e744d4aa86a20a74e2f94e4242669b87a7529c not found: ID does not exist" Nov 02 14:31:45.377682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.377600 4431 scope.go:122] "RemoveContainer" containerID="2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70" Nov 02 14:31:45.377942 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:31:45.377921 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70\": container with ID starting with 2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70 not found: ID does not exist" containerID="2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70" Nov 02 14:31:45.378001 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.377950 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70"} err="failed to get container status \"2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70\": rpc error: code = NotFound desc = could not find container \"2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70\": container with ID starting with 2637e2571fddfddff073eb20926df14976edafffabe76b971839c902e54c1e70 not found: ID does not exist" Nov 02 14:31:45.437206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.437177 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn"] Nov 02 14:31:45.450508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:45.450484 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d0c9b9-openshift-default-ffc94f6c9-4l9hn"] Nov 02 14:31:47.147826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:47.145950 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" path="/var/lib/kubelet/pods/fe2e3387-dee2-46b2-ad82-fcd52485a3aa/volumes" Nov 02 14:31:51.284865 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:31:51.284828 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:31:51.626043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.625921 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q"] Nov 02 14:31:51.626794 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.626725 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" Nov 02 14:31:51.626794 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.626750 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" Nov 02 14:31:51.627024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.626809 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" Nov 02 14:31:51.627024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.626819 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" Nov 02 14:31:51.627140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.627038 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" Nov 02 14:31:51.632317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.632294 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.649763 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.649743 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q"] Nov 02 14:31:51.658819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.658796 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e1c2\"/\"kube-root-ca.crt\"" Nov 02 14:31:51.658950 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.658823 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e1c2\"/\"openshift-service-ca.crt\"" Nov 02 14:31:51.661872 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.661854 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e1c2\"/\"default-dockercfg-8rtgf\"" Nov 02 14:31:51.661872 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.661855 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e1c2\"/\"rate-test-model-f1e1c2-kserve-self-signed-certs\"" Nov 02 14:31:51.728805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.728765 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-dshm\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.728964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.728819 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3534d-fc5b-4180-b613-6135efd7a12c-tls-certs\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.728964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.728885 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-tmp-dir\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.729038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.728974 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-model-cache\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.729089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.729064 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-home\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.729175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.729150 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmrh\" (UniqueName: \"kubernetes.io/projected/6bb3534d-fc5b-4180-b613-6135efd7a12c-kube-api-access-swmrh\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.830631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.830593 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-dshm\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.830834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.830652 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3534d-fc5b-4180-b613-6135efd7a12c-tls-certs\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.830834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.830693 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-tmp-dir\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.830834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.830764 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-model-cache\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.831035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.830893 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-home\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.831035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.830947 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmrh\" (UniqueName: \"kubernetes.io/projected/6bb3534d-fc5b-4180-b613-6135efd7a12c-kube-api-access-swmrh\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.831549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.831518 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-tmp-dir\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.831549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.831542 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-home\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.831760 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.831740 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-model-cache\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.833752 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.833721 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3534d-fc5b-4180-b613-6135efd7a12c-tls-certs\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.834081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.834061 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-dshm\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.883124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.878442 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmrh\" (UniqueName: \"kubernetes.io/projected/6bb3534d-fc5b-4180-b613-6135efd7a12c-kube-api-access-swmrh\") pod \"rate-test-model-f1e1c2-kserve-c85776749-75h6q\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:51.952805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:51.947710 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:31:52.275380 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:31:52.275333 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bb3534d_fc5b_4180_b613_6135efd7a12c.slice/crio-47d6ba696a34f77cea21f6398534dc5658f30eec108e7934fa0d15b2fb618507 WatchSource:0}: Error finding container 47d6ba696a34f77cea21f6398534dc5658f30eec108e7934fa0d15b2fb618507: Status 404 returned error can't find the container with id 47d6ba696a34f77cea21f6398534dc5658f30eec108e7934fa0d15b2fb618507 Nov 02 14:31:52.277087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:52.277058 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q"] Nov 02 14:31:52.412205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:52.412170 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" event={"ID":"6bb3534d-fc5b-4180-b613-6135efd7a12c","Type":"ContainerStarted","Data":"479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652"} Nov 02 14:31:52.412608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:52.412214 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" event={"ID":"6bb3534d-fc5b-4180-b613-6135efd7a12c","Type":"ContainerStarted","Data":"47d6ba696a34f77cea21f6398534dc5658f30eec108e7934fa0d15b2fb618507"} Nov 02 14:31:52.459873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:52.459827 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" podStartSLOduration=1.459812347 podStartE2EDuration="1.459812347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:52.455915108 +0000 UTC m=+2077.493020051" watchObservedRunningTime="2026-11-02 14:31:52.459812347 +0000 UTC m=+2077.496917281" Nov 02 14:31:52.948819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:31:52.948764 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:32:01.949055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:01.949016 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:32:01.962591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:01.962563 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:32:02.491295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:02.491264 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:32:12.566631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:12.566596 4431 generic.go:356] "Generic (PLEG): container finished" podID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerID="a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c" exitCode=0 Nov 02 14:32:12.567177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:12.566674 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" event={"ID":"3eb08832-f1bd-44e1-bf69-b9d3a298980b","Type":"ContainerDied","Data":"a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c"} Nov 02 14:32:12.567177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:12.567145 4431 scope.go:122] "RemoveContainer" containerID="a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c" Nov 02 14:32:12.568687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:12.568626 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:32:12.568808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:12.568693 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:32:12.568808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:12.568723 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:32:13.585806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:13.581963 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" event={"ID":"3eb08832-f1bd-44e1-bf69-b9d3a298980b","Type":"ContainerStarted","Data":"4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14"} Nov 02 14:32:14.429961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:14.429923 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:32:14.430141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:14.430104 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 14:32:14.430187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:14.430151 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 14:32:15.430271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:15.430238 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 14:32:15.430658 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:15.430298 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 14:32:16.429718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:16.429672 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 14:32:16.430078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:16.429735 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 14:32:17.430176 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:17.430134 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 14:32:17.430693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:17.430225 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 14:32:17.819344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:17.819265 4431 scope.go:122] "RemoveContainer" containerID="48e4cb0bbc02f6d2665e06072c81b363f81317d9d899d0830e7a6eb19dad5914" Nov 02 14:32:17.829335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:17.829310 4431 scope.go:122] "RemoveContainer" containerID="06c5ccebf6e6686fa681e6119ed7a5da95e825778a9172240b90c2e67429625b" Nov 02 14:32:18.430386 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.430344 4431 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body= Nov 02 14:32:18.430918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.430422 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" Nov 02 14:32:18.532730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.532669 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd"] Nov 02 14:32:18.535613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.534530 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe2e3387-dee2-46b2-ad82-fcd52485a3aa" containerName="istio-proxy" Nov 02 14:32:18.547938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.547918 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.569411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.569380 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-90a96161-openshift-default-dockercfg-68gwz\"" Nov 02 14:32:18.587826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.587013 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd"] Nov 02 14:32:18.679875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.679833 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-token\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.680078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.679883 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2sk6\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-kube-api-access-w2sk6\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.680078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.679995 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-socket\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.680078 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.680041 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-data\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.680267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.680175 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-credential-socket\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.680267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.680203 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-podinfo\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.680267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.680234 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istiod-ca-cert\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.680472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.680347 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-certs\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.680472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.680442 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-envoy\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782263 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782212 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-socket\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782281 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-data\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782408 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-credential-socket\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782452 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-podinfo\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782497 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istiod-ca-cert\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782531 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-certs\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782618 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-envoy\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782670 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-token\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.782937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.782699 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2sk6\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-kube-api-access-w2sk6\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.787796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.783796 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istiod-ca-cert\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.787796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.784165 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-socket\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.787796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.784343 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-certs\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.787796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.784436 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-credential-socket\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.787796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.784712 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-data\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.788204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.787987 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-envoy\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.817491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.817314 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-podinfo\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.820796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.818206 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2sk6\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-kube-api-access-w2sk6\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.823481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.823434 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-token\") pod \"e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:18.888928 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:18.888890 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:19.255434 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:32:19.255393 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aad9b7c_e120_4cb6_82dd_14a81f078af9.slice/crio-ec2398500c5be77fdb38340675f8019ddcbe6d6bf66c4cdd43fe3e9feab26345 WatchSource:0}: Error finding container ec2398500c5be77fdb38340675f8019ddcbe6d6bf66c4cdd43fe3e9feab26345: Status 404 returned error can't find the container with id ec2398500c5be77fdb38340675f8019ddcbe6d6bf66c4cdd43fe3e9feab26345 Nov 02 14:32:19.256245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.256218 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd"] Nov 02 14:32:19.258365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.258331 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:32:19.258488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.258396 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:32:19.258488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.258426 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:32:19.434326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.434297 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:32:19.434732 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.434715 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:32:19.435377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.435358 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:32:19.640231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.640199 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" event={"ID":"8aad9b7c-e120-4cb6-82dd-14a81f078af9","Type":"ContainerStarted","Data":"a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019"} Nov 02 14:32:19.640421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.640259 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" event={"ID":"8aad9b7c-e120-4cb6-82dd-14a81f078af9","Type":"ContainerStarted","Data":"ec2398500c5be77fdb38340675f8019ddcbe6d6bf66c4cdd43fe3e9feab26345"} Nov 02 14:32:19.700436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.700388 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" podStartSLOduration=1.70037171 podStartE2EDuration="1.70037171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:19.695317404 +0000 UTC m=+2104.732422339" watchObservedRunningTime="2026-11-02 14:32:19.70037171 +0000 UTC m=+2104.737476642" Nov 02 14:32:19.889174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:19.889136 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:20.895374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:20.895345 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:21.284935 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:32:21.284849 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:32:21.655260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:21.655219 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:21.656032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:21.656013 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:32:25.822173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:25.822135 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:32:33.264333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:33.264240 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8"] Nov 02 14:32:33.264838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:33.264494 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" containerID="cri-o://7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04" gracePeriod=30 Nov 02 14:32:37.081096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:37.081057 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr"] Nov 02 14:32:37.092094 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:37.092067 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17cc922d-ssrkr"] Nov 02 14:32:38.668178 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.668150 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:32:38.767677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.767644 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-certs\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.767971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.767719 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-credential-socket\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.767971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.767907 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-podinfo\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.767971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.767951 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-kube-api-access-dqxns\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-kube-api-access-dqxns\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.767971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.767954 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-certs" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:38.768338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.767989 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-socket\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.768338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.768076 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-credential-socket" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:38.768338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.768099 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-token\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-token\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.768338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.768135 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-envoy\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.768338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.768194 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04e0282-f36c-44cf-8a20-af39dcb9302c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e0282-f36c-44cf-8a20-af39dcb9302c-istiod-ca-cert\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.768338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.768240 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-data\") pod \"c04e0282-f36c-44cf-8a20-af39dcb9302c\" (UID: \"c04e0282-f36c-44cf-8a20-af39dcb9302c\") " Nov 02 14:32:38.768891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.768870 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:38.769021 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.768896 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:38.769161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.769125 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-data" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:38.769350 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.769329 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-socket" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:38.769936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.769909 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04e0282-f36c-44cf-8a20-af39dcb9302c-istiod-ca-cert" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:32:38.771757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.771728 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-podinfo" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:32:38.771897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.771829 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-envoy" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:32:38.772293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.772272 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-token" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:32:38.773532 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.773505 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-kube-api-access-dqxns" pod "c04e0282-f36c-44cf-8a20-af39dcb9302c" (UID: "c04e0282-f36c-44cf-8a20-af39dcb9302c"). InnerVolumeSpecName "kube-api-access-dqxns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:32:38.790250 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.790222 4431 generic.go:356] "Generic (PLEG): container finished" podID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerID="7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04" exitCode=0 Nov 02 14:32:38.790366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.790270 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" Nov 02 14:32:38.790366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.790299 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" event={"ID":"c04e0282-f36c-44cf-8a20-af39dcb9302c","Type":"ContainerDied","Data":"7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04"} Nov 02 14:32:38.790366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.790353 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8" event={"ID":"c04e0282-f36c-44cf-8a20-af39dcb9302c","Type":"ContainerDied","Data":"6419b338d60a7f531d2e35c22af486a0a4f53b07ab018a66f4ff8645ce0bb0f5"} Nov 02 14:32:38.790556 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.790380 4431 scope.go:122] "RemoveContainer" containerID="7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04" Nov 02 14:32:38.801646 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.801624 4431 scope.go:122] "RemoveContainer" containerID="ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692" Nov 02 14:32:38.815931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.814923 4431 scope.go:122] "RemoveContainer" containerID="7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04" Nov 02 14:32:38.815931 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:32:38.815346 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04\": container with ID starting with 7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04 not found: ID does not exist" containerID="7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04" Nov 02 14:32:38.815931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.815377 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04"} err="failed to get container status \"7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04\": rpc error: code = NotFound desc = could not find container \"7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04\": container with ID starting with 7f6ffa09e8347e722a3139ec64b210fb10e3c401a84f7857b127997643735f04 not found: ID does not exist" Nov 02 14:32:38.815931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.815397 4431 scope.go:122] "RemoveContainer" containerID="ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692" Nov 02 14:32:38.815931 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:32:38.815637 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692\": container with ID starting with ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692 not found: ID does not exist" containerID="ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692" Nov 02 14:32:38.815931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.815661 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692"} err="failed to get container status \"ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692\": rpc error: code = NotFound desc = could not find container \"ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692\": container with ID starting with ddf1495c30a55d8b484cf3758d58c2d04e56ab838eea383d680e5c4d9a6db692 not found: ID does not exist" Nov 02 14:32:38.853028 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.852976 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8"] Nov 02 14:32:38.858538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.858516 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-928a7a-openshift-default-6d557d6b67-d8hw8"] Nov 02 14:32:38.869863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.869842 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:38.869980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.869866 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxns\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-kube-api-access-dqxns\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:38.869980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.869895 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:38.869980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.869909 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:38.869980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.869921 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:38.869980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.869934 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04e0282-f36c-44cf-8a20-af39dcb9302c-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:38.869980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:38.869945 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04e0282-f36c-44cf-8a20-af39dcb9302c-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:32:39.076181 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:39.076152 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b" path="/var/lib/kubelet/pods/657b4bbb-d59f-4d70-b6df-d3e3a43d9b2b/volumes" Nov 02 14:32:39.076647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:39.076633 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" path="/var/lib/kubelet/pods/c04e0282-f36c-44cf-8a20-af39dcb9302c/volumes" Nov 02 14:32:48.052788 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:48.052741 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq"] Nov 02 14:32:48.065219 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:48.065195 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e36c10-kx9mq"] Nov 02 14:32:49.076517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:32:49.076485 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec36bd32-b429-4afc-91e2-8e07250e04e5" path="/var/lib/kubelet/pods/ec36bd32-b429-4afc-91e2-8e07250e04e5/volumes" Nov 02 14:32:51.283021 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:32:51.282985 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:33:04.040373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:04.040339 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp"] Nov 02 14:33:04.053394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:04.053366 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2e85a0-7nzhp"] Nov 02 14:33:04.861012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:04.860968 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:33:05.077032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:05.076999 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab3d2c5-e98a-467a-9c8d-945263033493" path="/var/lib/kubelet/pods/eab3d2c5-e98a-467a-9c8d-945263033493/volumes" Nov 02 14:33:07.574808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.574749 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4"] Nov 02 14:33:07.575306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.575287 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" Nov 02 14:33:07.575306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.575299 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" Nov 02 14:33:07.575437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.575420 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" Nov 02 14:33:07.586277 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.586234 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.599477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.599445 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4"] Nov 02 14:33:07.610671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.610382 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:33:07.610671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.610624 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:33:07.645474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.645428 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.645662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.645581 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dctm\" (UniqueName: \"kubernetes.io/projected/72bd6f9e-c912-4be3-9036-392335ea029c-kube-api-access-9dctm\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.645751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.645732 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.746918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.746889 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.747095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.746991 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dctm\" (UniqueName: \"kubernetes.io/projected/72bd6f9e-c912-4be3-9036-392335ea029c-kube-api-access-9dctm\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.747095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.747087 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.747770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.747735 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.749555 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.749527 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.772075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.772050 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dctm\" (UniqueName: \"kubernetes.io/projected/72bd6f9e-c912-4be3-9036-392335ea029c-kube-api-access-9dctm\") pod \"maas-api-revoke-keys-e2e-mt-778b2bce-9npg4\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:07.916211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:07.915929 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:08.266839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:08.266813 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4"] Nov 02 14:33:08.268242 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:33:08.268217 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bd6f9e_c912_4be3_9036_392335ea029c.slice/crio-55e21ac4e17a04c96786ff30644ac5c9ef2de8707ff69b594d8859527a806ec3 WatchSource:0}: Error finding container 55e21ac4e17a04c96786ff30644ac5c9ef2de8707ff69b594d8859527a806ec3: Status 404 returned error can't find the container with id 55e21ac4e17a04c96786ff30644ac5c9ef2de8707ff69b594d8859527a806ec3 Nov 02 14:33:09.007680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:09.007635 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" event={"ID":"72bd6f9e-c912-4be3-9036-392335ea029c","Type":"ContainerStarted","Data":"c506b2799404da09776c563e803aa6a1807f557d2c2674085b987d554980211c"} Nov 02 14:33:09.008191 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:09.007724 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" event={"ID":"72bd6f9e-c912-4be3-9036-392335ea029c","Type":"ContainerStarted","Data":"55e21ac4e17a04c96786ff30644ac5c9ef2de8707ff69b594d8859527a806ec3"} Nov 02 14:33:09.052473 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:09.052430 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" podStartSLOduration=2.052416483 podStartE2EDuration="2.052416483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:09.049952116 +0000 UTC m=+2154.087057048" watchObservedRunningTime="2026-11-02 14:33:09.052416483 +0000 UTC m=+2154.089521415" Nov 02 14:33:10.016241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:10.016203 4431 generic.go:356] "Generic (PLEG): container finished" podID="72bd6f9e-c912-4be3-9036-392335ea029c" containerID="c506b2799404da09776c563e803aa6a1807f557d2c2674085b987d554980211c" exitCode=0 Nov 02 14:33:10.016720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:10.016291 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" event={"ID":"72bd6f9e-c912-4be3-9036-392335ea029c","Type":"ContainerDied","Data":"c506b2799404da09776c563e803aa6a1807f557d2c2674085b987d554980211c"} Nov 02 14:33:11.184197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.184172 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:11.300878 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.300796 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-xks-ca\") pod \"72bd6f9e-c912-4be3-9036-392335ea029c\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " Nov 02 14:33:11.301055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.300893 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-openshift-ca\") pod \"72bd6f9e-c912-4be3-9036-392335ea029c\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " Nov 02 14:33:11.301055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.300936 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bd6f9e-c912-4be3-9036-392335ea029c-kube-api-access-9dctm\" (UniqueName: \"kubernetes.io/projected/72bd6f9e-c912-4be3-9036-392335ea029c-kube-api-access-9dctm\") pod \"72bd6f9e-c912-4be3-9036-392335ea029c\" (UID: \"72bd6f9e-c912-4be3-9036-392335ea029c\") " Nov 02 14:33:11.301236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.301204 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-openshift-ca" pod "72bd6f9e-c912-4be3-9036-392335ea029c" (UID: "72bd6f9e-c912-4be3-9036-392335ea029c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:33:11.301476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.301457 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:33:11.303012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.302989 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-xks-ca" pod "72bd6f9e-c912-4be3-9036-392335ea029c" (UID: "72bd6f9e-c912-4be3-9036-392335ea029c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:33:11.303089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.303013 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bd6f9e-c912-4be3-9036-392335ea029c-kube-api-access-9dctm" pod "72bd6f9e-c912-4be3-9036-392335ea029c" (UID: "72bd6f9e-c912-4be3-9036-392335ea029c"). InnerVolumeSpecName "kube-api-access-9dctm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:33:11.402832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.402807 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72bd6f9e-c912-4be3-9036-392335ea029c-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:33:11.402832 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:11.402832 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dctm\" (UniqueName: \"kubernetes.io/projected/72bd6f9e-c912-4be3-9036-392335ea029c-kube-api-access-9dctm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:33:12.034583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:12.034551 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" Nov 02 14:33:12.034583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:12.034583 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4" event={"ID":"72bd6f9e-c912-4be3-9036-392335ea029c","Type":"ContainerDied","Data":"55e21ac4e17a04c96786ff30644ac5c9ef2de8707ff69b594d8859527a806ec3"} Nov 02 14:33:12.034839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:12.034638 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e21ac4e17a04c96786ff30644ac5c9ef2de8707ff69b594d8859527a806ec3" Nov 02 14:33:15.039584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:15.039542 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4"] Nov 02 14:33:15.052381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:15.052350 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff2cd52-9pzv4"] Nov 02 14:33:15.076041 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:15.076008 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403f41ee-757a-4e26-836f-e8c454f7d039" path="/var/lib/kubelet/pods/403f41ee-757a-4e26-836f-e8c454f7d039/volumes" Nov 02 14:33:17.863139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:17.863104 4431 scope.go:122] "RemoveContainer" containerID="de64518cd1c50d7522658ea1c25a227932f687e7b16aab4994d15cefe0333207" Nov 02 14:33:21.286232 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:33:21.286190 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:33:23.031454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:23.031423 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf"] Nov 02 14:33:23.044058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:23.044032 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c86c866c-fs9nf"] Nov 02 14:33:23.098817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:23.098765 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a665063-5a9e-4f57-add5-817979f6f382" path="/var/lib/kubelet/pods/4a665063-5a9e-4f57-add5-817979f6f382/volumes" Nov 02 14:33:28.052301 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:28.052272 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl"] Nov 02 14:33:28.057619 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:28.057586 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf169b5f-c2bsl"] Nov 02 14:33:29.076639 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.076603 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66580ea1-2b59-4661-b1e1-4bdcb206dd9a" path="/var/lib/kubelet/pods/66580ea1-2b59-4661-b1e1-4bdcb206dd9a/volumes" Nov 02 14:33:29.233146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.233099 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582"] Nov 02 14:33:29.233645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.233626 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" Nov 02 14:33:29.233645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.233642 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" Nov 02 14:33:29.233879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.233659 4431 cpu_manager.go:414] "containerMap: removing container" podUID="72bd6f9e-c912-4be3-9036-392335ea029c" containerName="revoke-keys" Nov 02 14:33:29.233879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.233673 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bd6f9e-c912-4be3-9036-392335ea029c" containerName="revoke-keys" Nov 02 14:33:29.233879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.233803 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04e0282-f36c-44cf-8a20-af39dcb9302c" containerName="istio-proxy" Nov 02 14:33:29.233879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.233814 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bd6f9e-c912-4be3-9036-392335ea029c" containerName="revoke-keys" Nov 02 14:33:29.238839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.238819 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.258240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.258219 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582"] Nov 02 14:33:29.266934 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.266913 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:33:29.267050 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.266969 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:33:29.400333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.400294 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgf4w\" (UniqueName: \"kubernetes.io/projected/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-kube-api-access-vgf4w\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.400531 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.400512 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.400708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.400682 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.501682 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.501657 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.501855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.501744 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.501929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.501883 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgf4w\" (UniqueName: \"kubernetes.io/projected/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-kube-api-access-vgf4w\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.502322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.502301 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.504501 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.504476 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.537281 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.537255 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgf4w\" (UniqueName: \"kubernetes.io/projected/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-kube-api-access-vgf4w\") pod \"maas-api-revoke-sub-80b341aa-tn582\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.554142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.554118 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:29.858692 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:29.858667 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582"] Nov 02 14:33:30.170147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:30.170113 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" event={"ID":"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4","Type":"ContainerStarted","Data":"075d34962d6e65ab69523e39d07ada1ad47050d576b5719d84ad9a2e1c4f6f01"} Nov 02 14:33:31.180132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:31.180103 4431 generic.go:356] "Generic (PLEG): container finished" podID="0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4" containerID="335cbd6c98a9056078217e7e6303d5b3d754d5e3b8482c79cb9cb8f78a10aad7" exitCode=0 Nov 02 14:33:31.180569 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:31.180179 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" event={"ID":"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4","Type":"ContainerDied","Data":"335cbd6c98a9056078217e7e6303d5b3d754d5e3b8482c79cb9cb8f78a10aad7"} Nov 02 14:33:31.181952 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:31.181927 4431 generic.go:356] "Generic (PLEG): container finished" podID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerID="a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019" exitCode=0 Nov 02 14:33:31.182081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:31.182001 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" event={"ID":"8aad9b7c-e120-4cb6-82dd-14a81f078af9","Type":"ContainerDied","Data":"a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019"} Nov 02 14:33:31.182348 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:31.182334 4431 scope.go:122] "RemoveContainer" containerID="a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019" Nov 02 14:33:31.184052 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:31.184021 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:33:31.184138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:31.184103 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:33:31.184176 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:31.184148 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:33:32.216344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.216305 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" event={"ID":"8aad9b7c-e120-4cb6-82dd-14a81f078af9","Type":"ContainerStarted","Data":"f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181"} Nov 02 14:33:32.429260 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.429237 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:32.548506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.548433 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-kube-api-access-vgf4w\" (UniqueName: \"kubernetes.io/projected/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-kube-api-access-vgf4w\") pod \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " Nov 02 14:33:32.548506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.548475 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-xks-ca\") pod \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " Nov 02 14:33:32.548697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.548668 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-openshift-ca\") pod \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\" (UID: \"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4\") " Nov 02 14:33:32.549012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.548985 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-openshift-ca" pod "0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4" (UID: "0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:33:32.549305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.549286 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.550606 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.550579 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-xks-ca" pod "0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4" (UID: "0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:33:32.550697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.550654 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-kube-api-access-vgf4w" pod "0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4" (UID: "0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4"). InnerVolumeSpecName "kube-api-access-vgf4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:33:32.651175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.651136 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgf4w\" (UniqueName: \"kubernetes.io/projected/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-kube-api-access-vgf4w\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.651175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.651163 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:33:32.890186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.890161 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:33:32.895387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.895008 4431 patch_prober.go:29] interesting pod/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 02 14:33:32.895387 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:32.895060 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 02 14:33:33.227330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:33.227247 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" Nov 02 14:33:33.227330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:33.227262 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582" event={"ID":"0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4","Type":"ContainerDied","Data":"075d34962d6e65ab69523e39d07ada1ad47050d576b5719d84ad9a2e1c4f6f01"} Nov 02 14:33:33.227330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:33.227317 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="075d34962d6e65ab69523e39d07ada1ad47050d576b5719d84ad9a2e1c4f6f01" Nov 02 14:33:33.894802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:33.893842 4431 patch_prober.go:29] interesting pod/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 02 14:33:33.894802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:33.893909 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 02 14:33:34.890801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:34.890158 4431 patch_prober.go:29] interesting pod/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 02 14:33:34.890801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:34.890234 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 02 14:33:35.032865 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:35.032831 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt"] Nov 02 14:33:35.045890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:35.045860 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48fd8748-sqfkt"] Nov 02 14:33:35.099881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:35.099839 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa8f24c-0073-405a-be97-9575f08da07f" path="/var/lib/kubelet/pods/9fa8f24c-0073-405a-be97-9575f08da07f/volumes" Nov 02 14:33:35.894798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:35.892941 4431 patch_prober.go:29] interesting pod/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 02 14:33:35.894798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:35.892996 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 02 14:33:36.655529 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:36.655496 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:33:36.895122 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:36.895078 4431 patch_prober.go:29] interesting pod/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Nov 02 14:33:36.895617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:36.895158 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Nov 02 14:33:37.894451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:37.894423 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:33:37.895603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:37.895585 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:33:39.046880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:39.046849 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq"] Nov 02 14:33:39.048807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:39.048747 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b08f0db-qkcwq"] Nov 02 14:33:39.080874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:39.080847 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a285e2-6dc7-41e2-8057-0a411ab404df" path="/var/lib/kubelet/pods/14a285e2-6dc7-41e2-8057-0a411ab404df/volumes" Nov 02 14:33:43.937847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:43.937817 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:33:46.035907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:46.035868 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m"] Nov 02 14:33:46.047042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:46.047016 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6242810d-5zj6m"] Nov 02 14:33:47.076224 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:47.076191 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad23ad4-f234-4b62-8d57-42547fac459a" path="/var/lib/kubelet/pods/6ad23ad4-f234-4b62-8d57-42547fac459a/volumes" Nov 02 14:33:49.032984 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:49.032951 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g"] Nov 02 14:33:49.046987 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:49.046953 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33376e24-f726g"] Nov 02 14:33:49.076540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:49.076504 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba85d83-ea10-41e4-8507-e9fbd2bba1a1" path="/var/lib/kubelet/pods/dba85d83-ea10-41e4-8507-e9fbd2bba1a1/volumes" Nov 02 14:33:51.296656 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:33:51.296585 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:33:58.894614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:58.893520 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526"] Nov 02 14:33:58.901312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:33:58.895209 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" containerID="cri-o://0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748" gracePeriod=30 Nov 02 14:34:00.037446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:00.037411 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b"] Nov 02 14:34:00.050404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:00.050377 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753e7808-mvn9b"] Nov 02 14:34:01.076403 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:01.076370 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc48bfa2-967a-48e9-a521-f541ac888c8c" path="/var/lib/kubelet/pods/fc48bfa2-967a-48e9-a521-f541ac888c8c/volumes" Nov 02 14:34:04.168689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.168635 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:34:04.311704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.311673 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-envoy\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.311912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.311714 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-data\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.311912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.311848 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-certs\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.311912 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.311894 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-token\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-token\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.312149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312005 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-podinfo\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.312149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312081 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-data" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:34:04.312266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312147 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-credential-socket\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.312266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312181 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-socket\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.312266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312209 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istiod-ca-cert\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.312441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312271 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-kube-api-access-hqn7f\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-kube-api-access-hqn7f\") pod \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\" (UID: \"43cfed86-5e58-4292-8f84-4ee7ff3b5b76\") " Nov 02 14:34:04.312560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312202 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-certs" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:34:04.312560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312409 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-credential-socket" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:34:04.312677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312582 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-socket" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:34:04.312677 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312651 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istiod-ca-cert" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:34:04.312943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312919 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.313043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312948 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.313043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312963 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.313043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312976 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.313043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.312989 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.314244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.314219 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-podinfo" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:34:04.314422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.314397 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-envoy" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:34:04.314824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.314765 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-kube-api-access-hqn7f" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "kube-api-access-hqn7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:04.314936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.314860 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-token" pod "43cfed86-5e58-4292-8f84-4ee7ff3b5b76" (UID: "43cfed86-5e58-4292-8f84-4ee7ff3b5b76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:04.414395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.414368 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.414395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.414391 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.414560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.414402 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqn7f\" (UniqueName: \"kubernetes.io/projected/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-kube-api-access-hqn7f\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.414560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.414411 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfed86-5e58-4292-8f84-4ee7ff3b5b76-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:04.487587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.487520 4431 generic.go:356] "Generic (PLEG): container finished" podID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerID="0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748" exitCode=0 Nov 02 14:34:04.487718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.487587 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" Nov 02 14:34:04.487718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.487645 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" event={"ID":"43cfed86-5e58-4292-8f84-4ee7ff3b5b76","Type":"ContainerDied","Data":"0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748"} Nov 02 14:34:04.487882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.487746 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526" event={"ID":"43cfed86-5e58-4292-8f84-4ee7ff3b5b76","Type":"ContainerDied","Data":"94eccbe73ad626eff036d4f73494fc4fe33410031f7ee5ba760622205b701359"} Nov 02 14:34:04.487882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.487790 4431 scope.go:122] "RemoveContainer" containerID="0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748" Nov 02 14:34:04.499716 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.499697 4431 scope.go:122] "RemoveContainer" containerID="0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748" Nov 02 14:34:04.500012 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:34:04.499993 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748\": container with ID starting with 0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748 not found: ID does not exist" containerID="0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748" Nov 02 14:34:04.500087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.500020 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748"} err="failed to get container status \"0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748\": rpc error: code = NotFound desc = could not find container \"0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748\": container with ID starting with 0dbf4d7a1946a7e19b3a8fcd9c9c8f1f2b5990f08a3fef97d2b81bf1c2ef9748 not found: ID does not exist" Nov 02 14:34:04.544197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.544166 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526"] Nov 02 14:34:04.557852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:04.557833 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-12b8da-openshift-default-7df679ddc7-wg526"] Nov 02 14:34:05.078227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:05.078191 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" path="/var/lib/kubelet/pods/43cfed86-5e58-4292-8f84-4ee7ff3b5b76/volumes" Nov 02 14:34:07.447817 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:07.447771 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4"] Nov 02 14:34:07.460915 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:07.460883 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-778b2bce-9npg4"] Nov 02 14:34:09.078720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:09.078677 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bd6f9e-c912-4be3-9036-392335ea029c" path="/var/lib/kubelet/pods/72bd6f9e-c912-4be3-9036-392335ea029c/volumes" Nov 02 14:34:12.031568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:12.031527 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm"] Nov 02 14:34:12.044481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:12.044451 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25361b0a-44jfm"] Nov 02 14:34:13.083307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:13.083269 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4ef40c-226c-4edc-b775-d71291e4ca90" path="/var/lib/kubelet/pods/ef4ef40c-226c-4edc-b775-d71291e4ca90/volumes" Nov 02 14:34:17.895340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:17.895300 4431 scope.go:122] "RemoveContainer" containerID="f8f7a3557b1ba3dac573b8ad380398ad46407b9bfdde6a43cad9b3f11e42aa43" Nov 02 14:34:17.905412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:17.905391 4431 scope.go:122] "RemoveContainer" containerID="b7fadf0aba84d0d8b3229680053270ec5d44bb1d8ae2ec038f8e1f8787e54889" Nov 02 14:34:17.913879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:17.913854 4431 scope.go:122] "RemoveContainer" containerID="b296f0e4bdd990d20d76c3191c1d5411fdd12b6e54835218623ba43fdd003c04" Nov 02 14:34:17.922203 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:17.922182 4431 scope.go:122] "RemoveContainer" containerID="996c62e4d26cc1b227e4fc209af5db04081d56579379426c328e0aad714bc364" Nov 02 14:34:21.282851 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:34:21.282766 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:34:27.636306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:27.632328 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:34:31.971099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.971060 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28"] Nov 02 14:34:31.971795 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.971757 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4" containerName="revoke-keys" Nov 02 14:34:31.971919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.971795 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4" containerName="revoke-keys" Nov 02 14:34:31.971919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.971848 4431 cpu_manager.go:414] "containerMap: removing container" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" Nov 02 14:34:31.971919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.971857 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" Nov 02 14:34:31.972086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.972018 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cfed86-5e58-4292-8f84-4ee7ff3b5b76" containerName="istio-proxy" Nov 02 14:34:31.972086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.972040 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4" containerName="revoke-keys" Nov 02 14:34:31.976697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.976675 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:31.999512 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:31.999481 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28"] Nov 02 14:34:32.008640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.008621 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:34:32.008890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.008873 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:34:32.080641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.080610 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.080806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.080757 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.080923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.080902 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjrzg\" (UniqueName: \"kubernetes.io/projected/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-kube-api-access-qjrzg\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.182483 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.182455 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.182631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.182549 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.182631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.182614 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjrzg\" (UniqueName: \"kubernetes.io/projected/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-kube-api-access-qjrzg\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.183276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.183254 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.184940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.184916 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.210281 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.210254 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjrzg\" (UniqueName: \"kubernetes.io/projected/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-kube-api-access-qjrzg\") pod \"maas-api-revoke-sub-6cab08d4-mqf28\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.289690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.289632 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:32.592493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.592467 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28"] Nov 02 14:34:32.593817 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:34:32.593789 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dfb9c1_b5bb_486d_9c8a_0a48b54ac8fd.slice/crio-bd4edf47438700afe90fbeeda1beb74d49e6170ad4a1d39a5243f124efe80540 WatchSource:0}: Error finding container bd4edf47438700afe90fbeeda1beb74d49e6170ad4a1d39a5243f124efe80540: Status 404 returned error can't find the container with id bd4edf47438700afe90fbeeda1beb74d49e6170ad4a1d39a5243f124efe80540 Nov 02 14:34:32.722101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:32.722073 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" event={"ID":"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd","Type":"ContainerStarted","Data":"bd4edf47438700afe90fbeeda1beb74d49e6170ad4a1d39a5243f124efe80540"} Nov 02 14:34:33.729503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:33.729473 4431 generic.go:356] "Generic (PLEG): container finished" podID="98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd" containerID="1b782092e82e7c7d6376b546889b0d97783b6cd803cfcb33c061691e0b9e9806" exitCode=0 Nov 02 14:34:33.729909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:33.729569 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" event={"ID":"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd","Type":"ContainerDied","Data":"1b782092e82e7c7d6376b546889b0d97783b6cd803cfcb33c061691e0b9e9806"} Nov 02 14:34:34.900920 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:34.900892 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:35.015162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.015086 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-kube-api-access-qjrzg\" (UniqueName: \"kubernetes.io/projected/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-kube-api-access-qjrzg\") pod \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " Nov 02 14:34:35.015306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.015181 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-xks-ca\") pod \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " Nov 02 14:34:35.015440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.015421 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-openshift-ca\") pod \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\" (UID: \"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd\") " Nov 02 14:34:35.015813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.015762 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-openshift-ca" pod "98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd" (UID: "98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:34:35.017194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.017166 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-kube-api-access-qjrzg" pod "98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd" (UID: "98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd"). InnerVolumeSpecName "kube-api-access-qjrzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:35.017286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.017215 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-xks-ca" pod "98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd" (UID: "98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:34:35.117299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.117276 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:35.117299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.117298 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjrzg\" (UniqueName: \"kubernetes.io/projected/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-kube-api-access-qjrzg\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:35.117428 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.117308 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:35.746851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.746817 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" event={"ID":"98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd","Type":"ContainerDied","Data":"bd4edf47438700afe90fbeeda1beb74d49e6170ad4a1d39a5243f124efe80540"} Nov 02 14:34:35.746851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.746834 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28" Nov 02 14:34:35.747099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:35.746865 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd4edf47438700afe90fbeeda1beb74d49e6170ad4a1d39a5243f124efe80540" Nov 02 14:34:42.312443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.312412 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b"] Nov 02 14:34:42.313008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.312988 4431 cpu_manager.go:414] "containerMap: removing container" podUID="98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd" containerName="revoke-keys" Nov 02 14:34:42.313059 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.313009 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd" containerName="revoke-keys" Nov 02 14:34:42.313094 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.313085 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd" containerName="revoke-keys" Nov 02 14:34:42.316359 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.316341 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.341183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.341158 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b"] Nov 02 14:34:42.353812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.351031 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:34:42.353812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.351125 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:34:42.411789 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.411748 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.411941 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.411878 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.412060 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.412039 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbcst\" (UniqueName: \"kubernetes.io/projected/1df89671-e602-4498-a23f-eeb99eff061c-kube-api-access-dbcst\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.513085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.513050 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.513234 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.513214 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.513415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.513381 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcst\" (UniqueName: \"kubernetes.io/projected/1df89671-e602-4498-a23f-eeb99eff061c-kube-api-access-dbcst\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.513920 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.513893 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.515741 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.515716 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.540909 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.539997 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbcst\" (UniqueName: \"kubernetes.io/projected/1df89671-e602-4498-a23f-eeb99eff061c-kube-api-access-dbcst\") pod \"maas-api-revoke-sub-a2fa91b2-rfp7b\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.631022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.630951 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:42.935005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:42.934976 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b"] Nov 02 14:34:42.936486 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:34:42.936460 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df89671_e602_4498_a23f_eeb99eff061c.slice/crio-2b338eb9a143c6d59db568b1a2c56363f66d24479f843f08b579117b032be173 WatchSource:0}: Error finding container 2b338eb9a143c6d59db568b1a2c56363f66d24479f843f08b579117b032be173: Status 404 returned error can't find the container with id 2b338eb9a143c6d59db568b1a2c56363f66d24479f843f08b579117b032be173 Nov 02 14:34:43.801378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:43.801341 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" event={"ID":"1df89671-e602-4498-a23f-eeb99eff061c","Type":"ContainerStarted","Data":"a702ff0eaeae29d927c099cf5e03c50fc4c835458ce4a46ab45c4282fa4ddc96"} Nov 02 14:34:43.801761 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:43.801400 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" event={"ID":"1df89671-e602-4498-a23f-eeb99eff061c","Type":"ContainerStarted","Data":"2b338eb9a143c6d59db568b1a2c56363f66d24479f843f08b579117b032be173"} Nov 02 14:34:43.845485 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:43.845440 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" podStartSLOduration=1.845428503 podStartE2EDuration="1.845428503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:43.844362609 +0000 UTC m=+2248.881467541" watchObservedRunningTime="2026-11-02 14:34:43.845428503 +0000 UTC m=+2248.882533434" Nov 02 14:34:44.809901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:44.809866 4431 generic.go:356] "Generic (PLEG): container finished" podID="1df89671-e602-4498-a23f-eeb99eff061c" containerID="a702ff0eaeae29d927c099cf5e03c50fc4c835458ce4a46ab45c4282fa4ddc96" exitCode=0 Nov 02 14:34:44.810329 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:44.809959 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" event={"ID":"1df89671-e602-4498-a23f-eeb99eff061c","Type":"ContainerDied","Data":"a702ff0eaeae29d927c099cf5e03c50fc4c835458ce4a46ab45c4282fa4ddc96"} Nov 02 14:34:45.971354 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:45.971330 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:46.055135 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.055106 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df89671-e602-4498-a23f-eeb99eff061c-kube-api-access-dbcst\" (UniqueName: \"kubernetes.io/projected/1df89671-e602-4498-a23f-eeb99eff061c-kube-api-access-dbcst\") pod \"1df89671-e602-4498-a23f-eeb99eff061c\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " Nov 02 14:34:46.055311 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.055217 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-openshift-ca\") pod \"1df89671-e602-4498-a23f-eeb99eff061c\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " Nov 02 14:34:46.055383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.055349 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-xks-ca\") pod \"1df89671-e602-4498-a23f-eeb99eff061c\" (UID: \"1df89671-e602-4498-a23f-eeb99eff061c\") " Nov 02 14:34:46.055635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.055608 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-openshift-ca" pod "1df89671-e602-4498-a23f-eeb99eff061c" (UID: "1df89671-e602-4498-a23f-eeb99eff061c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:34:46.055966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.055945 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:46.057248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.057223 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-xks-ca" pod "1df89671-e602-4498-a23f-eeb99eff061c" (UID: "1df89671-e602-4498-a23f-eeb99eff061c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:34:46.057707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.057683 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df89671-e602-4498-a23f-eeb99eff061c-kube-api-access-dbcst" pod "1df89671-e602-4498-a23f-eeb99eff061c" (UID: "1df89671-e602-4498-a23f-eeb99eff061c"). InnerVolumeSpecName "kube-api-access-dbcst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:34:46.157678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.157650 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df89671-e602-4498-a23f-eeb99eff061c-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:46.157852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.157684 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbcst\" (UniqueName: \"kubernetes.io/projected/1df89671-e602-4498-a23f-eeb99eff061c-kube-api-access-dbcst\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:34:46.827661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.827636 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" Nov 02 14:34:46.827908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.827639 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b" event={"ID":"1df89671-e602-4498-a23f-eeb99eff061c","Type":"ContainerDied","Data":"2b338eb9a143c6d59db568b1a2c56363f66d24479f843f08b579117b032be173"} Nov 02 14:34:46.827908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:46.827759 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b338eb9a143c6d59db568b1a2c56363f66d24479f843f08b579117b032be173" Nov 02 14:34:51.282929 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:34:51.282890 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:34:56.567417 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:34:56.567365 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:35:01.361458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.361427 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v"] Nov 02 14:35:01.361988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.361974 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1df89671-e602-4498-a23f-eeb99eff061c" containerName="revoke-keys" Nov 02 14:35:01.362042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.361988 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df89671-e602-4498-a23f-eeb99eff061c" containerName="revoke-keys" Nov 02 14:35:01.362100 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.362091 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df89671-e602-4498-a23f-eeb99eff061c" containerName="revoke-keys" Nov 02 14:35:01.365532 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.365513 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.389321 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.389289 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v"] Nov 02 14:35:01.396793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.396758 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:35:01.396951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.396895 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:35:01.536427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.536393 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.536427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.536425 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.536672 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.536615 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pxf\" (UniqueName: \"kubernetes.io/projected/756424eb-a5bd-444f-8408-dad53396ee4f-kube-api-access-29pxf\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.637599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.637574 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.637742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.637609 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.637814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.637770 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pxf\" (UniqueName: \"kubernetes.io/projected/756424eb-a5bd-444f-8408-dad53396ee4f-kube-api-access-29pxf\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.638332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.638303 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.639901 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.639877 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.667451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.667421 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pxf\" (UniqueName: \"kubernetes.io/projected/756424eb-a5bd-444f-8408-dad53396ee4f-kube-api-access-29pxf\") pod \"maas-api-revoke-sub-7ce61dad-bj85v\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:01.679243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:01.679224 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:02.003494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:02.003464 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v"] Nov 02 14:35:02.004896 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:35:02.004871 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756424eb_a5bd_444f_8408_dad53396ee4f.slice/crio-7022ab9d274bbc19c5971c48e985d6cf172f0714af832ed5dc7634d1b7ebe1b2 WatchSource:0}: Error finding container 7022ab9d274bbc19c5971c48e985d6cf172f0714af832ed5dc7634d1b7ebe1b2: Status 404 returned error can't find the container with id 7022ab9d274bbc19c5971c48e985d6cf172f0714af832ed5dc7634d1b7ebe1b2 Nov 02 14:35:02.934689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:02.934654 4431 generic.go:356] "Generic (PLEG): container finished" podID="756424eb-a5bd-444f-8408-dad53396ee4f" containerID="c80f98f327a8b375faa265045625f77531d695574718ef236bec3ee09a10a27e" exitCode=0 Nov 02 14:35:02.935064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:02.934706 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" event={"ID":"756424eb-a5bd-444f-8408-dad53396ee4f","Type":"ContainerDied","Data":"c80f98f327a8b375faa265045625f77531d695574718ef236bec3ee09a10a27e"} Nov 02 14:35:02.935064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:02.934761 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" event={"ID":"756424eb-a5bd-444f-8408-dad53396ee4f","Type":"ContainerStarted","Data":"7022ab9d274bbc19c5971c48e985d6cf172f0714af832ed5dc7634d1b7ebe1b2"} Nov 02 14:35:04.094707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.094682 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:04.272117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.272025 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756424eb-a5bd-444f-8408-dad53396ee4f-kube-api-access-29pxf\" (UniqueName: \"kubernetes.io/projected/756424eb-a5bd-444f-8408-dad53396ee4f-kube-api-access-29pxf\") pod \"756424eb-a5bd-444f-8408-dad53396ee4f\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " Nov 02 14:35:04.272117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.272106 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-xks-ca\") pod \"756424eb-a5bd-444f-8408-dad53396ee4f\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " Nov 02 14:35:04.272363 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.272274 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-openshift-ca\") pod \"756424eb-a5bd-444f-8408-dad53396ee4f\" (UID: \"756424eb-a5bd-444f-8408-dad53396ee4f\") " Nov 02 14:35:04.272666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.272644 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-openshift-ca" pod "756424eb-a5bd-444f-8408-dad53396ee4f" (UID: "756424eb-a5bd-444f-8408-dad53396ee4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:04.272898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.272873 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:04.274229 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.274206 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756424eb-a5bd-444f-8408-dad53396ee4f-kube-api-access-29pxf" pod "756424eb-a5bd-444f-8408-dad53396ee4f" (UID: "756424eb-a5bd-444f-8408-dad53396ee4f"). InnerVolumeSpecName "kube-api-access-29pxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:04.274308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.274288 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-xks-ca" pod "756424eb-a5bd-444f-8408-dad53396ee4f" (UID: "756424eb-a5bd-444f-8408-dad53396ee4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:35:04.373828 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.373768 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29pxf\" (UniqueName: \"kubernetes.io/projected/756424eb-a5bd-444f-8408-dad53396ee4f-kube-api-access-29pxf\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:04.373828 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.373831 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756424eb-a5bd-444f-8408-dad53396ee4f-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:04.948304 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.948273 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" Nov 02 14:35:04.948495 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.948311 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v" event={"ID":"756424eb-a5bd-444f-8408-dad53396ee4f","Type":"ContainerDied","Data":"7022ab9d274bbc19c5971c48e985d6cf172f0714af832ed5dc7634d1b7ebe1b2"} Nov 02 14:35:04.948495 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:04.948360 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7022ab9d274bbc19c5971c48e985d6cf172f0714af832ed5dc7634d1b7ebe1b2" Nov 02 14:35:13.349841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.349802 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n"] Nov 02 14:35:13.350536 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.350514 4431 cpu_manager.go:414] "containerMap: removing container" podUID="756424eb-a5bd-444f-8408-dad53396ee4f" containerName="revoke-keys" Nov 02 14:35:13.350671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.350537 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756424eb-a5bd-444f-8408-dad53396ee4f" containerName="revoke-keys" Nov 02 14:35:13.350739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.350718 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="756424eb-a5bd-444f-8408-dad53396ee4f" containerName="revoke-keys" Nov 02 14:35:13.353947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.353925 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.370954 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.370929 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:35:13.375822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.375795 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.375966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.375947 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh94b\" (UniqueName: \"kubernetes.io/projected/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-kube-api-access-hh94b\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.376027 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.376002 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.384255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.384233 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:35:13.387893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.387872 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n"] Nov 02 14:35:13.477305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.477277 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh94b\" (UniqueName: \"kubernetes.io/projected/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-kube-api-access-hh94b\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.477427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.477324 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.477427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.477409 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.478123 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.478086 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.480506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.480481 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.502683 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.502658 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh94b\" (UniqueName: \"kubernetes.io/projected/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-kube-api-access-hh94b\") pod \"maas-api-revoke-sub-7069a045-gpv8n\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.667904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.667880 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:13.975594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:13.975566 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n"] Nov 02 14:35:13.976875 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:35:13.976848 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fd92b6_8de4_4322_ae07_d21439ce7e6a.slice/crio-b549faf1f529b2ef0f4577e89bf167e09c4b0098756a0726313a4baef895f32e WatchSource:0}: Error finding container b549faf1f529b2ef0f4577e89bf167e09c4b0098756a0726313a4baef895f32e: Status 404 returned error can't find the container with id b549faf1f529b2ef0f4577e89bf167e09c4b0098756a0726313a4baef895f32e Nov 02 14:35:14.008703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:14.008677 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" event={"ID":"e5fd92b6-8de4-4322-ae07-d21439ce7e6a","Type":"ContainerStarted","Data":"b549faf1f529b2ef0f4577e89bf167e09c4b0098756a0726313a4baef895f32e"} Nov 02 14:35:15.017171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:15.017141 4431 generic.go:356] "Generic (PLEG): container finished" podID="e5fd92b6-8de4-4322-ae07-d21439ce7e6a" containerID="c8377756965d23bc14a09e8fe91f59494353ab3649100043013abbf3702a487b" exitCode=0 Nov 02 14:35:15.017527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:15.017229 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" event={"ID":"e5fd92b6-8de4-4322-ae07-d21439ce7e6a","Type":"ContainerDied","Data":"c8377756965d23bc14a09e8fe91f59494353ab3649100043013abbf3702a487b"} Nov 02 14:35:16.187557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.187533 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:16.309266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.309203 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-openshift-ca\") pod \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " Nov 02 14:35:16.309425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.309385 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-xks-ca\") pod \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " Nov 02 14:35:16.309425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.309414 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-kube-api-access-hh94b\" (UniqueName: \"kubernetes.io/projected/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-kube-api-access-hh94b\") pod \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\" (UID: \"e5fd92b6-8de4-4322-ae07-d21439ce7e6a\") " Nov 02 14:35:16.309605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.309582 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-openshift-ca" pod "e5fd92b6-8de4-4322-ae07-d21439ce7e6a" (UID: "e5fd92b6-8de4-4322-ae07-d21439ce7e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:35:16.309943 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.309924 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:16.311458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.311429 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-xks-ca" pod "e5fd92b6-8de4-4322-ae07-d21439ce7e6a" (UID: "e5fd92b6-8de4-4322-ae07-d21439ce7e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:35:16.311558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.311463 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-kube-api-access-hh94b" pod "e5fd92b6-8de4-4322-ae07-d21439ce7e6a" (UID: "e5fd92b6-8de4-4322-ae07-d21439ce7e6a"). InnerVolumeSpecName "kube-api-access-hh94b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:16.411409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.411385 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:16.411409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:16.411408 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh94b\" (UniqueName: \"kubernetes.io/projected/e5fd92b6-8de4-4322-ae07-d21439ce7e6a-kube-api-access-hh94b\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:17.033453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:17.033420 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" Nov 02 14:35:17.033651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:17.033419 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n" event={"ID":"e5fd92b6-8de4-4322-ae07-d21439ce7e6a","Type":"ContainerDied","Data":"b549faf1f529b2ef0f4577e89bf167e09c4b0098756a0726313a4baef895f32e"} Nov 02 14:35:17.033651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:17.033570 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b549faf1f529b2ef0f4577e89bf167e09c4b0098756a0726313a4baef895f32e" Nov 02 14:35:17.967397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:17.967354 4431 scope.go:122] "RemoveContainer" containerID="69d05f9110bce0c52a6ed4cad913a9c50d28974e52a9726ec88495d516802342" Nov 02 14:35:17.977265 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:17.977234 4431 scope.go:122] "RemoveContainer" containerID="e8703b5a15baabd7c8f2126a547eca7ff91397709513ae1887d3dc2c80a23836" Nov 02 14:35:17.986888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:17.986869 4431 scope.go:122] "RemoveContainer" containerID="2eea8f0b71b577fed291e2dda8907d0682512decc2a976ecaacbb28346a7350e" Nov 02 14:35:17.995131 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:17.995112 4431 scope.go:122] "RemoveContainer" containerID="d73da1a04c423e53c84e3659961d8d16a5db2231fce4450f35b856060ceda4dc" Nov 02 14:35:18.003885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:18.003862 4431 scope.go:122] "RemoveContainer" containerID="c59d5a6c0bff762113f44abd907e4a9983e8bf3d61e29d1af9a7312cc29f9263" Nov 02 14:35:18.013068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:18.013050 4431 scope.go:122] "RemoveContainer" containerID="9f1cc4897174ecdf47b8474ec6cde55cdb577e0b89965a0c57979f82052f4d5f" Nov 02 14:35:18.021083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:18.021067 4431 scope.go:122] "RemoveContainer" containerID="a13488aa2f5c2dbc2068b5e7eae21895a5af981b2af0236d272df6f1412e639b" Nov 02 14:35:18.029237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:18.029221 4431 scope.go:122] "RemoveContainer" containerID="39060a90914e61871438c701a3114b1a09703d9a6701ec9cc5bac969ba117010" Nov 02 14:35:18.037551 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:18.037535 4431 scope.go:122] "RemoveContainer" containerID="d6170728c05a1829f108b0da3b42ff633914b24d77e84827783f3db2ca82dc67" Nov 02 14:35:19.562491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.562452 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7807-4143b6"] Nov 02 14:35:19.563249 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.563227 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e5fd92b6-8de4-4322-ae07-d21439ce7e6a" containerName="revoke-keys" Nov 02 14:35:19.563249 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.563249 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fd92b6-8de4-4322-ae07-d21439ce7e6a" containerName="revoke-keys" Nov 02 14:35:19.563423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.563394 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fd92b6-8de4-4322-ae07-d21439ce7e6a" containerName="revoke-keys" Nov 02 14:35:19.568721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.568700 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-4143b6" Nov 02 14:35:19.584413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.584390 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7807-4143b6"] Nov 02 14:35:19.593988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.593971 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nz2z6\"" Nov 02 14:35:19.656960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.656934 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbrkk\" (UniqueName: \"kubernetes.io/projected/3eab138b-53f7-44b4-bf53-08f38e5671ad-kube-api-access-sbrkk\") pod \"test-curl-7807-4143b6\" (UID: \"3eab138b-53f7-44b4-bf53-08f38e5671ad\") " pod="opendatahub/test-curl-7807-4143b6" Nov 02 14:35:19.758019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.757984 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbrkk\" (UniqueName: \"kubernetes.io/projected/3eab138b-53f7-44b4-bf53-08f38e5671ad-kube-api-access-sbrkk\") pod \"test-curl-7807-4143b6\" (UID: \"3eab138b-53f7-44b4-bf53-08f38e5671ad\") " pod="opendatahub/test-curl-7807-4143b6" Nov 02 14:35:19.783248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.783227 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbrkk\" (UniqueName: \"kubernetes.io/projected/3eab138b-53f7-44b4-bf53-08f38e5671ad-kube-api-access-sbrkk\") pod \"test-curl-7807-4143b6\" (UID: \"3eab138b-53f7-44b4-bf53-08f38e5671ad\") " pod="opendatahub/test-curl-7807-4143b6" Nov 02 14:35:19.881495 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:19.881443 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-4143b6" Nov 02 14:35:20.188747 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:20.188724 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7807-4143b6"] Nov 02 14:35:20.190237 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:35:20.190214 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eab138b_53f7_44b4_bf53_08f38e5671ad.slice/crio-07af7f93ee2eb4005638b3f7fac8004adc98ae936b80a53203a478deacd71e23 WatchSource:0}: Error finding container 07af7f93ee2eb4005638b3f7fac8004adc98ae936b80a53203a478deacd71e23: Status 404 returned error can't find the container with id 07af7f93ee2eb4005638b3f7fac8004adc98ae936b80a53203a478deacd71e23 Nov 02 14:35:21.067261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:21.067223 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7807-4143b6" event={"ID":"3eab138b-53f7-44b4-bf53-08f38e5671ad","Type":"ContainerStarted","Data":"4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0"} Nov 02 14:35:21.067656 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:21.067268 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7807-4143b6" event={"ID":"3eab138b-53f7-44b4-bf53-08f38e5671ad","Type":"ContainerStarted","Data":"07af7f93ee2eb4005638b3f7fac8004adc98ae936b80a53203a478deacd71e23"} Nov 02 14:35:21.117267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:21.117215 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7807-4143b6" podStartSLOduration=2.117200591 podStartE2EDuration="2.117200591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:21.115377288 +0000 UTC m=+2286.152482220" watchObservedRunningTime="2026-11-02 14:35:21.117200591 +0000 UTC m=+2286.154305523" Nov 02 14:35:21.285849 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:35:21.285809 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:35:21.993006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:21.992968 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7807-4143b6"] Nov 02 14:35:21.998627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:21.998599 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7807-4143b6"] Nov 02 14:35:23.080993 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:23.080932 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7807-4143b6" podUID="3eab138b-53f7-44b4-bf53-08f38e5671ad" containerName="test-curl-7807-4143b6" containerID="cri-o://4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0" gracePeriod=2 Nov 02 14:35:25.302673 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:25.302648 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-4143b6" Nov 02 14:35:25.431940 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:25.431910 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eab138b-53f7-44b4-bf53-08f38e5671ad-kube-api-access-sbrkk\" (UniqueName: \"kubernetes.io/projected/3eab138b-53f7-44b4-bf53-08f38e5671ad-kube-api-access-sbrkk\") pod \"3eab138b-53f7-44b4-bf53-08f38e5671ad\" (UID: \"3eab138b-53f7-44b4-bf53-08f38e5671ad\") " Nov 02 14:35:25.434129 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:25.434101 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eab138b-53f7-44b4-bf53-08f38e5671ad-kube-api-access-sbrkk" pod "3eab138b-53f7-44b4-bf53-08f38e5671ad" (UID: "3eab138b-53f7-44b4-bf53-08f38e5671ad"). InnerVolumeSpecName "kube-api-access-sbrkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:25.533465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:25.533441 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbrkk\" (UniqueName: \"kubernetes.io/projected/3eab138b-53f7-44b4-bf53-08f38e5671ad-kube-api-access-sbrkk\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:26.103592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:26.103555 4431 generic.go:356] "Generic (PLEG): container finished" podID="3eab138b-53f7-44b4-bf53-08f38e5671ad" containerID="4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0" exitCode=137 Nov 02 14:35:26.103592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:26.103592 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7807-4143b6" Nov 02 14:35:26.103859 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:26.103653 4431 scope.go:122] "RemoveContainer" containerID="4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0" Nov 02 14:35:26.115267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:26.115236 4431 scope.go:122] "RemoveContainer" containerID="4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0" Nov 02 14:35:26.118925 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:35:26.118497 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0\": container with ID starting with 4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0 not found: ID does not exist" containerID="4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0" Nov 02 14:35:26.118925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:26.118540 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0"} err="failed to get container status \"4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0\": rpc error: code = NotFound desc = could not find container \"4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0\": container with ID starting with 4640fd178c4a49e57957e91e83e0372a2835a828a048b70c0ed1cc6b0867a5b0 not found: ID does not exist" Nov 02 14:35:26.129078 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:35:26.129049 4431 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7807-4143b6\" is forbidden: User \"system:node:ip-10-0-134-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-9.ec2.internal' and this object" podUID="3eab138b-53f7-44b4-bf53-08f38e5671ad" pod="opendatahub/test-curl-7807-4143b6" Nov 02 14:35:27.076332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:27.076299 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eab138b-53f7-44b4-bf53-08f38e5671ad" path="/var/lib/kubelet/pods/3eab138b-53f7-44b4-bf53-08f38e5671ad/volumes" Nov 02 14:35:27.498230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:27.498196 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds"] Nov 02 14:35:30.033235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:30.033207 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb"] Nov 02 14:35:30.046296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:30.046266 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489a8abe-cjbvb"] Nov 02 14:35:31.076449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:31.076414 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f2609e-f1ba-4cce-9b7e-afbbf672473c" path="/var/lib/kubelet/pods/b5f2609e-f1ba-4cce-9b7e-afbbf672473c/volumes" Nov 02 14:35:40.740206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:40.740170 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:35:42.519729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.519671 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" podUID="09604db3-ce99-474f-9448-d937b4461323" containerName="main" containerID="cri-o://04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2" gracePeriod=45 Nov 02 14:35:42.795173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.795148 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:35:42.845505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.845475 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-dshm\") pod \"09604db3-ce99-474f-9448-d937b4461323\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " Nov 02 14:35:42.845675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.845521 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-tmp-dir\") pod \"09604db3-ce99-474f-9448-d937b4461323\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " Nov 02 14:35:42.845675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.845590 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09604db3-ce99-474f-9448-d937b4461323-tls-certs\" (UniqueName: \"kubernetes.io/secret/09604db3-ce99-474f-9448-d937b4461323-tls-certs\") pod \"09604db3-ce99-474f-9448-d937b4461323\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " Nov 02 14:35:42.845820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.845746 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-home\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-home\") pod \"09604db3-ce99-474f-9448-d937b4461323\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " Nov 02 14:35:42.845820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.845799 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09604db3-ce99-474f-9448-d937b4461323-kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/09604db3-ce99-474f-9448-d937b4461323-kube-api-access-jtwjz\") pod \"09604db3-ce99-474f-9448-d937b4461323\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " Nov 02 14:35:42.845933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.845846 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-tmp-dir" pod "09604db3-ce99-474f-9448-d937b4461323" (UID: "09604db3-ce99-474f-9448-d937b4461323"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:35:42.845933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.845868 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-model-cache\") pod \"09604db3-ce99-474f-9448-d937b4461323\" (UID: \"09604db3-ce99-474f-9448-d937b4461323\") " Nov 02 14:35:42.846077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.846056 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-home" pod "09604db3-ce99-474f-9448-d937b4461323" (UID: "09604db3-ce99-474f-9448-d937b4461323"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:35:42.846256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.846236 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-model-cache" pod "09604db3-ce99-474f-9448-d937b4461323" (UID: "09604db3-ce99-474f-9448-d937b4461323"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:35:42.846434 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.846419 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:42.846503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.846435 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:42.846503 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.846446 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:42.848089 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.848058 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09604db3-ce99-474f-9448-d937b4461323-tls-certs" pod "09604db3-ce99-474f-9448-d937b4461323" (UID: "09604db3-ce99-474f-9448-d937b4461323"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:35:42.848193 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.848168 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09604db3-ce99-474f-9448-d937b4461323-kube-api-access-jtwjz" pod "09604db3-ce99-474f-9448-d937b4461323" (UID: "09604db3-ce99-474f-9448-d937b4461323"). InnerVolumeSpecName "kube-api-access-jtwjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:35:42.848241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.848202 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-dshm" pod "09604db3-ce99-474f-9448-d937b4461323" (UID: "09604db3-ce99-474f-9448-d937b4461323"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:35:42.947893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.947857 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/09604db3-ce99-474f-9448-d937b4461323-kube-api-access-jtwjz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:42.947893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.947890 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09604db3-ce99-474f-9448-d937b4461323-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:42.947893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:42.947899 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09604db3-ce99-474f-9448-d937b4461323-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:35:43.210352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.210325 4431 generic.go:356] "Generic (PLEG): container finished" podID="09604db3-ce99-474f-9448-d937b4461323" containerID="04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2" exitCode=0 Nov 02 14:35:43.210502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.210373 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" Nov 02 14:35:43.210502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.210390 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" event={"ID":"09604db3-ce99-474f-9448-d937b4461323","Type":"ContainerDied","Data":"04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2"} Nov 02 14:35:43.210502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.210430 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds" event={"ID":"09604db3-ce99-474f-9448-d937b4461323","Type":"ContainerDied","Data":"ced99a589cb77af97857ff321a1be978f4d0f31e549fa8164d2f989295bfe55d"} Nov 02 14:35:43.210502 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.210451 4431 scope.go:122] "RemoveContainer" containerID="04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2" Nov 02 14:35:43.220298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.220280 4431 scope.go:122] "RemoveContainer" containerID="04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2" Nov 02 14:35:43.220552 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:35:43.220532 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2\": container with ID starting with 04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2 not found: ID does not exist" containerID="04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2" Nov 02 14:35:43.220607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.220559 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2"} err="failed to get container status \"04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2\": rpc error: code = NotFound desc = could not find container \"04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2\": container with ID starting with 04ef7b815638562ce9c9a48a50fb78acfefa7a78b26ecbb16d798d8bfad5ded2 not found: ID does not exist" Nov 02 14:35:43.268429 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.268400 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds"] Nov 02 14:35:43.281408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:43.281376 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-778b2bce/e2e-lifecycle-model-778b2bce-kserve-676884f589-qrdds"] Nov 02 14:35:45.076828 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:45.076795 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09604db3-ce99-474f-9448-d937b4461323" path="/var/lib/kubelet/pods/09604db3-ce99-474f-9448-d937b4461323/volumes" Nov 02 14:35:51.284535 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:35:51.284493 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:35:52.032927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:52.032890 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5"] Nov 02 14:35:52.045770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:52.045744 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b1bd63e-vzdc5"] Nov 02 14:35:53.075546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:35:53.075512 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d644dc-5b60-4624-8fdb-7a05b09504b1" path="/var/lib/kubelet/pods/61d644dc-5b60-4624-8fdb-7a05b09504b1/volumes" Nov 02 14:36:01.320552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.320518 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn"] Nov 02 14:36:01.321095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.321080 4431 cpu_manager.go:414] "containerMap: removing container" podUID="09604db3-ce99-474f-9448-d937b4461323" containerName="main" Nov 02 14:36:01.321142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.321096 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09604db3-ce99-474f-9448-d937b4461323" containerName="main" Nov 02 14:36:01.321142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.321132 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3eab138b-53f7-44b4-bf53-08f38e5671ad" containerName="test-curl-7807-4143b6" Nov 02 14:36:01.321142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.321140 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eab138b-53f7-44b4-bf53-08f38e5671ad" containerName="test-curl-7807-4143b6" Nov 02 14:36:01.321248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.321217 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="09604db3-ce99-474f-9448-d937b4461323" containerName="main" Nov 02 14:36:01.321248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.321236 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eab138b-53f7-44b4-bf53-08f38e5671ad" containerName="test-curl-7807-4143b6" Nov 02 14:36:01.326032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.326012 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.355918 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.355890 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:36:01.356056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.355941 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:36:01.358947 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.358922 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn"] Nov 02 14:36:01.465287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.465255 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7v2\" (UniqueName: \"kubernetes.io/projected/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-kube-api-access-hq7v2\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.465438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.465296 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.465496 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.465474 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.566345 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.566312 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.566491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.566386 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7v2\" (UniqueName: \"kubernetes.io/projected/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-kube-api-access-hq7v2\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.566491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.566414 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.567063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.567045 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.568652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.568633 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.590894 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.590838 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7v2\" (UniqueName: \"kubernetes.io/projected/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-kube-api-access-hq7v2\") pod \"maas-api-revoke-sub-25d67b4b-27tdn\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.638700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.638674 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:01.942266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:01.942235 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn"] Nov 02 14:36:01.943228 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:36:01.943202 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b27e3b_f0ad_44ce_8bf6_ce8ed08ab195.slice/crio-a97340672a06c17e90955b97c049b4c5f24c5944bfe7440f28d33478b921e34a WatchSource:0}: Error finding container a97340672a06c17e90955b97c049b4c5f24c5944bfe7440f28d33478b921e34a: Status 404 returned error can't find the container with id a97340672a06c17e90955b97c049b4c5f24c5944bfe7440f28d33478b921e34a Nov 02 14:36:02.373453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:02.373418 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" event={"ID":"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195","Type":"ContainerStarted","Data":"b9bfa330a183e0a6744b5a0b0a6ef0bf388a3403fc4425c7b715d0be916442ca"} Nov 02 14:36:02.374023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:02.373483 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" event={"ID":"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195","Type":"ContainerStarted","Data":"a97340672a06c17e90955b97c049b4c5f24c5944bfe7440f28d33478b921e34a"} Nov 02 14:36:02.419787 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:02.419726 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" podStartSLOduration=1.419712307 podStartE2EDuration="1.419712307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:02.417207332 +0000 UTC m=+2327.454312264" watchObservedRunningTime="2026-11-02 14:36:02.419712307 +0000 UTC m=+2327.456817238" Nov 02 14:36:03.389448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:03.389409 4431 generic.go:356] "Generic (PLEG): container finished" podID="20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195" containerID="b9bfa330a183e0a6744b5a0b0a6ef0bf388a3403fc4425c7b715d0be916442ca" exitCode=0 Nov 02 14:36:03.389863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:03.389500 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" event={"ID":"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195","Type":"ContainerDied","Data":"b9bfa330a183e0a6744b5a0b0a6ef0bf388a3403fc4425c7b715d0be916442ca"} Nov 02 14:36:04.547655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.547632 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:04.624413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.624378 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-xks-ca\") pod \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " Nov 02 14:36:04.624585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.624458 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-kube-api-access-hq7v2\" (UniqueName: \"kubernetes.io/projected/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-kube-api-access-hq7v2\") pod \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " Nov 02 14:36:04.624585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.624496 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-openshift-ca\") pod \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\" (UID: \"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195\") " Nov 02 14:36:04.624905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.624883 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-openshift-ca" pod "20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195" (UID: "20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:04.625083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.625052 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:04.626460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.626438 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-xks-ca" pod "20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195" (UID: "20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:04.626604 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.626590 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-kube-api-access-hq7v2" pod "20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195" (UID: "20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195"). InnerVolumeSpecName "kube-api-access-hq7v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:04.726253 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.726192 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:04.726253 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:04.726216 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq7v2\" (UniqueName: \"kubernetes.io/projected/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195-kube-api-access-hq7v2\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:05.404227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:05.404199 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" Nov 02 14:36:05.406572 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:05.406533 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn" event={"ID":"20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195","Type":"ContainerDied","Data":"a97340672a06c17e90955b97c049b4c5f24c5944bfe7440f28d33478b921e34a"} Nov 02 14:36:05.406704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:05.406597 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a97340672a06c17e90955b97c049b4c5f24c5944bfe7440f28d33478b921e34a" Nov 02 14:36:11.670130 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.670091 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h"] Nov 02 14:36:11.670723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.670706 4431 cpu_manager.go:414] "containerMap: removing container" podUID="20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195" containerName="revoke-keys" Nov 02 14:36:11.670723 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.670724 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195" containerName="revoke-keys" Nov 02 14:36:11.670856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.670846 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195" containerName="revoke-keys" Nov 02 14:36:11.674042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.674025 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.703416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.703394 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:36:11.703416 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.703408 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:36:11.706581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.706560 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h"] Nov 02 14:36:11.719481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.719454 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8r7z\" (UniqueName: \"kubernetes.io/projected/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-kube-api-access-q8r7z\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.719623 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.719550 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.719707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.719683 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.820566 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.820535 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8r7z\" (UniqueName: \"kubernetes.io/projected/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-kube-api-access-q8r7z\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.820718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.820595 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.820718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.820647 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.821310 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.821286 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.823096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.823074 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.844783 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.844756 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8r7z\" (UniqueName: \"kubernetes.io/projected/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-kube-api-access-q8r7z\") pod \"maas-api-revoke-sub-50cd8266-47c5h\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:11.987552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:11.987465 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:12.289338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:12.289315 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h"] Nov 02 14:36:12.290970 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:36:12.290937 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a9bfd5_53c0_4a58_ad92_57d03f6d2710.slice/crio-34cda43dfcda6ff47291151842a3f220338233a12dcc3cf409c3dfdca573d62b WatchSource:0}: Error finding container 34cda43dfcda6ff47291151842a3f220338233a12dcc3cf409c3dfdca573d62b: Status 404 returned error can't find the container with id 34cda43dfcda6ff47291151842a3f220338233a12dcc3cf409c3dfdca573d62b Nov 02 14:36:12.465991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:12.465956 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" event={"ID":"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710","Type":"ContainerStarted","Data":"34cda43dfcda6ff47291151842a3f220338233a12dcc3cf409c3dfdca573d62b"} Nov 02 14:36:13.474452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:13.474411 4431 generic.go:356] "Generic (PLEG): container finished" podID="a9a9bfd5-53c0-4a58-ad92-57d03f6d2710" containerID="d14563e3836e117a508aedb577602906434c09bc0ba2457f13820228238b3d6e" exitCode=0 Nov 02 14:36:13.474961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:13.474623 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" event={"ID":"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710","Type":"ContainerDied","Data":"d14563e3836e117a508aedb577602906434c09bc0ba2457f13820228238b3d6e"} Nov 02 14:36:14.630038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.630015 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:14.775095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.775018 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-kube-api-access-q8r7z\" (UniqueName: \"kubernetes.io/projected/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-kube-api-access-q8r7z\") pod \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " Nov 02 14:36:14.775095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.775073 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-xks-ca\") pod \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " Nov 02 14:36:14.775329 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.775246 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-openshift-ca\") pod \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\" (UID: \"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710\") " Nov 02 14:36:14.775679 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.775653 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-openshift-ca" pod "a9a9bfd5-53c0-4a58-ad92-57d03f6d2710" (UID: "a9a9bfd5-53c0-4a58-ad92-57d03f6d2710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:14.775888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.775868 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:14.777230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.777198 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-xks-ca" pod "a9a9bfd5-53c0-4a58-ad92-57d03f6d2710" (UID: "a9a9bfd5-53c0-4a58-ad92-57d03f6d2710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:14.777333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.777265 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-kube-api-access-q8r7z" pod "a9a9bfd5-53c0-4a58-ad92-57d03f6d2710" (UID: "a9a9bfd5-53c0-4a58-ad92-57d03f6d2710"). InnerVolumeSpecName "kube-api-access-q8r7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:14.876540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.876510 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8r7z\" (UniqueName: \"kubernetes.io/projected/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-kube-api-access-q8r7z\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:14.876540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:14.876538 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:15.490330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:15.490285 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" event={"ID":"a9a9bfd5-53c0-4a58-ad92-57d03f6d2710","Type":"ContainerDied","Data":"34cda43dfcda6ff47291151842a3f220338233a12dcc3cf409c3dfdca573d62b"} Nov 02 14:36:15.490330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:15.490312 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h" Nov 02 14:36:15.490611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:15.490347 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34cda43dfcda6ff47291151842a3f220338233a12dcc3cf409c3dfdca573d62b" Nov 02 14:36:21.284222 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:36:21.284185 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:36:22.514081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.514047 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn"] Nov 02 14:36:22.515082 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.515042 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" containerID="cri-o://a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b" gracePeriod=30 Nov 02 14:36:22.688075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.687990 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7"] Nov 02 14:36:22.689141 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.689115 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a9a9bfd5-53c0-4a58-ad92-57d03f6d2710" containerName="revoke-keys" Nov 02 14:36:22.689313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.689299 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a9bfd5-53c0-4a58-ad92-57d03f6d2710" containerName="revoke-keys" Nov 02 14:36:22.689729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.689709 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a9bfd5-53c0-4a58-ad92-57d03f6d2710" containerName="revoke-keys" Nov 02 14:36:22.694975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.694954 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.714409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.714380 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:36:22.722645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.719176 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7"] Nov 02 14:36:22.723065 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.723026 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:36:22.805149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.805086 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.805149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.805113 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9sjj\" (UniqueName: \"kubernetes.io/projected/01e48e3a-374f-45e1-aac5-f26c9b89c55c-kube-api-access-r9sjj\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.805312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.805268 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.906136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.906110 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.906261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.906217 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.906261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.906236 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9sjj\" (UniqueName: \"kubernetes.io/projected/01e48e3a-374f-45e1-aac5-f26c9b89c55c-kube-api-access-r9sjj\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.906850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.906821 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.908372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.908351 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:22.933636 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:22.933610 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9sjj\" (UniqueName: \"kubernetes.io/projected/01e48e3a-374f-45e1-aac5-f26c9b89c55c-kube-api-access-r9sjj\") pod \"maas-api-revoke-sub-9f6f3ec1-j2nf7\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:23.016881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:23.016853 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:23.321376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:23.321345 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7"] Nov 02 14:36:23.322304 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:36:23.322277 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e48e3a_374f_45e1_aac5_f26c9b89c55c.slice/crio-b1c4e88b8949a9f16e2bdb2a61991cd7cec55e86b34db38951a6dd54b760c856 WatchSource:0}: Error finding container b1c4e88b8949a9f16e2bdb2a61991cd7cec55e86b34db38951a6dd54b760c856: Status 404 returned error can't find the container with id b1c4e88b8949a9f16e2bdb2a61991cd7cec55e86b34db38951a6dd54b760c856 Nov 02 14:36:23.569501 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:23.569421 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" event={"ID":"01e48e3a-374f-45e1-aac5-f26c9b89c55c","Type":"ContainerStarted","Data":"b1c4e88b8949a9f16e2bdb2a61991cd7cec55e86b34db38951a6dd54b760c856"} Nov 02 14:36:24.178544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.178510 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94"] Nov 02 14:36:24.182727 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.182709 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.204973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.204947 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94"] Nov 02 14:36:24.223814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.223787 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvrv\" (UniqueName: \"kubernetes.io/projected/1f2875bf-aabf-4099-b0be-390554272184-kube-api-access-ksvrv\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.223994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.223974 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2875bf-aabf-4099-b0be-390554272184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.224188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.224172 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2875bf-aabf-4099-b0be-390554272184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.325458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.325428 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvrv\" (UniqueName: \"kubernetes.io/projected/1f2875bf-aabf-4099-b0be-390554272184-kube-api-access-ksvrv\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.325605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.325488 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2875bf-aabf-4099-b0be-390554272184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.325605 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.325596 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2875bf-aabf-4099-b0be-390554272184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.326242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.326220 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2875bf-aabf-4099-b0be-390554272184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.327873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.327854 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2875bf-aabf-4099-b0be-390554272184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.354739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.354685 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvrv\" (UniqueName: \"kubernetes.io/projected/1f2875bf-aabf-4099-b0be-390554272184-kube-api-access-ksvrv\") pod \"maas-api-revoke-sub-a6fb08d5-csn94\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.495840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.495812 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:24.582055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.582023 4431 generic.go:356] "Generic (PLEG): container finished" podID="01e48e3a-374f-45e1-aac5-f26c9b89c55c" containerID="a68f408d8d2fddf9063ba9b211869228a44361e9ebf80477de9e6e81423fb2bd" exitCode=0 Nov 02 14:36:24.582451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.582105 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" event={"ID":"01e48e3a-374f-45e1-aac5-f26c9b89c55c","Type":"ContainerDied","Data":"a68f408d8d2fddf9063ba9b211869228a44361e9ebf80477de9e6e81423fb2bd"} Nov 02 14:36:24.804264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:24.804237 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94"] Nov 02 14:36:24.805799 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:36:24.805742 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2875bf_aabf_4099_b0be_390554272184.slice/crio-b8c3200a5362a0dc99164fce15435861fd646991a4be161ef0c2ea1214fc76cb WatchSource:0}: Error finding container b8c3200a5362a0dc99164fce15435861fd646991a4be161ef0c2ea1214fc76cb: Status 404 returned error can't find the container with id b8c3200a5362a0dc99164fce15435861fd646991a4be161ef0c2ea1214fc76cb Nov 02 14:36:25.333842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.333729 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:36:25.589904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.589827 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" event={"ID":"1f2875bf-aabf-4099-b0be-390554272184","Type":"ContainerStarted","Data":"afee97c6cc078814bf21097634cbaccef4cff019d73d8eff4169c8bb0c25468a"} Nov 02 14:36:25.589904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.589887 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" event={"ID":"1f2875bf-aabf-4099-b0be-390554272184","Type":"ContainerStarted","Data":"b8c3200a5362a0dc99164fce15435861fd646991a4be161ef0c2ea1214fc76cb"} Nov 02 14:36:25.633650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.633598 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" podStartSLOduration=1.633584633 podStartE2EDuration="1.633584633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:25.632768302 +0000 UTC m=+2350.669873234" watchObservedRunningTime="2026-11-02 14:36:25.633584633 +0000 UTC m=+2350.670689564" Nov 02 14:36:25.837508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.837482 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:25.981305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.981275 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-xks-ca\") pod \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " Nov 02 14:36:25.981491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.981316 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e48e3a-374f-45e1-aac5-f26c9b89c55c-kube-api-access-r9sjj\" (UniqueName: \"kubernetes.io/projected/01e48e3a-374f-45e1-aac5-f26c9b89c55c-kube-api-access-r9sjj\") pod \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " Nov 02 14:36:25.981491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.981349 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-openshift-ca\") pod \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\" (UID: \"01e48e3a-374f-45e1-aac5-f26c9b89c55c\") " Nov 02 14:36:25.981737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.981713 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-openshift-ca" pod "01e48e3a-374f-45e1-aac5-f26c9b89c55c" (UID: "01e48e3a-374f-45e1-aac5-f26c9b89c55c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:25.981923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.981906 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:25.983542 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.983520 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-xks-ca" pod "01e48e3a-374f-45e1-aac5-f26c9b89c55c" (UID: "01e48e3a-374f-45e1-aac5-f26c9b89c55c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:25.983666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:25.983608 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e48e3a-374f-45e1-aac5-f26c9b89c55c-kube-api-access-r9sjj" pod "01e48e3a-374f-45e1-aac5-f26c9b89c55c" (UID: "01e48e3a-374f-45e1-aac5-f26c9b89c55c"). InnerVolumeSpecName "kube-api-access-r9sjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:26.082617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.082592 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e48e3a-374f-45e1-aac5-f26c9b89c55c-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:26.082617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.082614 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9sjj\" (UniqueName: \"kubernetes.io/projected/01e48e3a-374f-45e1-aac5-f26c9b89c55c-kube-api-access-r9sjj\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:26.498690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.498653 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx"] Nov 02 14:36:26.499171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.499142 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" containerID="cri-o://e3dd0ce0de42a37ed1ecd91115c486fb9e0aae1bb8c03fbad6864e30559a67ca" gracePeriod=30 Nov 02 14:36:26.511814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.511765 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l"] Nov 02 14:36:26.512056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.512016 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" containerID="cri-o://a41e333a41a2841c798878e3432d12c4619efe4934e743a23c5e49b9ca2f5e41" gracePeriod=30 Nov 02 14:36:26.515017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.514993 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5"] Nov 02 14:36:26.515248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.515209 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" containerID="cri-o://01f1a807ee3f63eacc4ff59eac55f49a6cec4662c9d8106df8d6f194d0f4ff37" gracePeriod=30 Nov 02 14:36:26.525515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.525488 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc"] Nov 02 14:36:26.525703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.525674 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" containerID="cri-o://d1e3a8568c79cff14bb00cbd2d2c2c243f785f218e9f97ef861111dfe8d280ab" gracePeriod=30 Nov 02 14:36:26.599242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.599205 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" event={"ID":"01e48e3a-374f-45e1-aac5-f26c9b89c55c","Type":"ContainerDied","Data":"b1c4e88b8949a9f16e2bdb2a61991cd7cec55e86b34db38951a6dd54b760c856"} Nov 02 14:36:26.599758 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.599261 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1c4e88b8949a9f16e2bdb2a61991cd7cec55e86b34db38951a6dd54b760c856" Nov 02 14:36:26.599758 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.599216 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7" Nov 02 14:36:26.601183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.601156 4431 generic.go:356] "Generic (PLEG): container finished" podID="1f2875bf-aabf-4099-b0be-390554272184" containerID="afee97c6cc078814bf21097634cbaccef4cff019d73d8eff4169c8bb0c25468a" exitCode=0 Nov 02 14:36:26.601308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:26.601214 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" event={"ID":"1f2875bf-aabf-4099-b0be-390554272184","Type":"ContainerDied","Data":"afee97c6cc078814bf21097634cbaccef4cff019d73d8eff4169c8bb0c25468a"} Nov 02 14:36:27.786024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.785963 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:27.824971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.824949 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:36:27.906822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.906761 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-certs\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.906958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.906891 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-credential-socket\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.906958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.906939 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-data\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.907030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.906975 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2875bf-aabf-4099-b0be-390554272184-kube-api-access-ksvrv\" (UniqueName: \"kubernetes.io/projected/1f2875bf-aabf-4099-b0be-390554272184-kube-api-access-ksvrv\") pod \"1f2875bf-aabf-4099-b0be-390554272184\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " Nov 02 14:36:27.907030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.906983 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-certs" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:27.907030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907014 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istiod-ca-cert\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.907177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907115 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-token\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-token\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.907177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907142 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-podinfo\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.907286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907217 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2875bf-aabf-4099-b0be-390554272184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2875bf-aabf-4099-b0be-390554272184-maas-api-openshift-ca\") pod \"1f2875bf-aabf-4099-b0be-390554272184\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " Nov 02 14:36:27.907286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907258 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-socket\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.907406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907299 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2875bf-aabf-4099-b0be-390554272184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2875bf-aabf-4099-b0be-390554272184-maas-api-xks-ca\") pod \"1f2875bf-aabf-4099-b0be-390554272184\" (UID: \"1f2875bf-aabf-4099-b0be-390554272184\") " Nov 02 14:36:27.907406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907213 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-credential-socket" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:27.907406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907226 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-data" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:27.907406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907329 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-kube-api-access-v7xl4\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-kube-api-access-v7xl4\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.907630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.907452 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-envoy\") pod \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\" (UID: \"61d580e7-f40b-47ea-b707-e1c6c3e144ac\") " Nov 02 14:36:27.908058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.908032 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:27.908058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.908037 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-socket" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:27.908262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.908062 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:27.908262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.908077 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:27.908449 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.908426 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istiod-ca-cert" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:27.908640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.908610 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2875bf-aabf-4099-b0be-390554272184-maas-api-openshift-ca" pod "1f2875bf-aabf-4099-b0be-390554272184" (UID: "1f2875bf-aabf-4099-b0be-390554272184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:27.909737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.909712 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-podinfo" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:36:27.909849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.909758 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2875bf-aabf-4099-b0be-390554272184-kube-api-access-ksvrv" pod "1f2875bf-aabf-4099-b0be-390554272184" (UID: "1f2875bf-aabf-4099-b0be-390554272184"). InnerVolumeSpecName "kube-api-access-ksvrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:27.910056 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.910035 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-token" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:27.910340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.910318 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2875bf-aabf-4099-b0be-390554272184-maas-api-xks-ca" pod "1f2875bf-aabf-4099-b0be-390554272184" (UID: "1f2875bf-aabf-4099-b0be-390554272184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:27.910539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.910520 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-envoy" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:27.910884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:27.910863 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-kube-api-access-v7xl4" pod "61d580e7-f40b-47ea-b707-e1c6c3e144ac" (UID: "61d580e7-f40b-47ea-b707-e1c6c3e144ac"). InnerVolumeSpecName "kube-api-access-v7xl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:28.009306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009283 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.009306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009306 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvrv\" (UniqueName: \"kubernetes.io/projected/1f2875bf-aabf-4099-b0be-390554272184-kube-api-access-ksvrv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.009443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009315 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.009443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009329 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.009443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009339 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d580e7-f40b-47ea-b707-e1c6c3e144ac-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.009443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009347 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f2875bf-aabf-4099-b0be-390554272184-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.009443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009355 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d580e7-f40b-47ea-b707-e1c6c3e144ac-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.009443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009363 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f2875bf-aabf-4099-b0be-390554272184-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.009443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.009370 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xl4\" (UniqueName: \"kubernetes.io/projected/61d580e7-f40b-47ea-b707-e1c6c3e144ac-kube-api-access-v7xl4\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:28.617064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.617036 4431 generic.go:356] "Generic (PLEG): container finished" podID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerID="a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b" exitCode=0 Nov 02 14:36:28.617198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.617081 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" Nov 02 14:36:28.617198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.617124 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" event={"ID":"61d580e7-f40b-47ea-b707-e1c6c3e144ac","Type":"ContainerDied","Data":"a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b"} Nov 02 14:36:28.617198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.617174 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn" event={"ID":"61d580e7-f40b-47ea-b707-e1c6c3e144ac","Type":"ContainerDied","Data":"7b6498866bc6faec4d4b8e17de0644d1e7c092035fdf9e10266f71819d793e77"} Nov 02 14:36:28.617386 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.617204 4431 scope.go:122] "RemoveContainer" containerID="a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b" Nov 02 14:36:28.618907 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.618887 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" event={"ID":"1f2875bf-aabf-4099-b0be-390554272184","Type":"ContainerDied","Data":"b8c3200a5362a0dc99164fce15435861fd646991a4be161ef0c2ea1214fc76cb"} Nov 02 14:36:28.619022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.618919 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94" Nov 02 14:36:28.619022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.618930 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c3200a5362a0dc99164fce15435861fd646991a4be161ef0c2ea1214fc76cb" Nov 02 14:36:28.628736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.628718 4431 scope.go:122] "RemoveContainer" containerID="1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376" Nov 02 14:36:28.638223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.638209 4431 scope.go:122] "RemoveContainer" containerID="a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b" Nov 02 14:36:28.638478 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:36:28.638457 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b\": container with ID starting with a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b not found: ID does not exist" containerID="a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b" Nov 02 14:36:28.638516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.638488 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b"} err="failed to get container status \"a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b\": rpc error: code = NotFound desc = could not find container \"a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b\": container with ID starting with a69179c187798d36ca4fe2543acd58cfe745d39dbc59f724b43bfe89e1dd945b not found: ID does not exist" Nov 02 14:36:28.638516 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.638504 4431 scope.go:122] "RemoveContainer" containerID="1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376" Nov 02 14:36:28.638751 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:36:28.638728 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376\": container with ID starting with 1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376 not found: ID does not exist" containerID="1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376" Nov 02 14:36:28.638840 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.638761 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376"} err="failed to get container status \"1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376\": rpc error: code = NotFound desc = could not find container \"1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376\": container with ID starting with 1f0f0a6ba9a3931931eb7023611aba42058321216d66afc4ed824d10a7069376 not found: ID does not exist" Nov 02 14:36:28.709409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.709386 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn"] Nov 02 14:36:28.713334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:28.713314 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-778b2bce-openshift-default-7685bc8b9c-h76xn"] Nov 02 14:36:29.075493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:29.075460 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" path="/var/lib/kubelet/pods/61d580e7-f40b-47ea-b707-e1c6c3e144ac/volumes" Nov 02 14:36:30.048139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:30.048099 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-pxflc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:36:30.048307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:30.048166 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:36:30.076565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:30.076531 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-jbt2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:36:30.076955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:30.076582 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:36:31.644374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.644340 4431 generic.go:356] "Generic (PLEG): container finished" podID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerID="01f1a807ee3f63eacc4ff59eac55f49a6cec4662c9d8106df8d6f194d0f4ff37" exitCode=0 Nov 02 14:36:31.644770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.644467 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" event={"ID":"09f2d68a-4f50-4fec-afed-5a20268f857c","Type":"ContainerDied","Data":"01f1a807ee3f63eacc4ff59eac55f49a6cec4662c9d8106df8d6f194d0f4ff37"} Nov 02 14:36:31.646691 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.646667 4431 generic.go:356] "Generic (PLEG): container finished" podID="1927399f-2905-4061-8966-8a5d2d98cbda" containerID="d1e3a8568c79cff14bb00cbd2d2c2c243f785f218e9f97ef861111dfe8d280ab" exitCode=0 Nov 02 14:36:31.646824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.646721 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" event={"ID":"1927399f-2905-4061-8966-8a5d2d98cbda","Type":"ContainerDied","Data":"d1e3a8568c79cff14bb00cbd2d2c2c243f785f218e9f97ef861111dfe8d280ab"} Nov 02 14:36:31.649683 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.649654 4431 generic.go:356] "Generic (PLEG): container finished" podID="e0881c94-9060-442e-b479-17013ddeb3ca" containerID="a41e333a41a2841c798878e3432d12c4619efe4934e743a23c5e49b9ca2f5e41" exitCode=0 Nov 02 14:36:31.649876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.649750 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" event={"ID":"e0881c94-9060-442e-b479-17013ddeb3ca","Type":"ContainerDied","Data":"a41e333a41a2841c798878e3432d12c4619efe4934e743a23c5e49b9ca2f5e41"} Nov 02 14:36:31.652302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.652276 4431 generic.go:356] "Generic (PLEG): container finished" podID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerID="e3dd0ce0de42a37ed1ecd91115c486fb9e0aae1bb8c03fbad6864e30559a67ca" exitCode=0 Nov 02 14:36:31.652506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.652457 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" event={"ID":"5152eb70-3989-4a4d-a6e7-5df967b974ac","Type":"ContainerDied","Data":"e3dd0ce0de42a37ed1ecd91115c486fb9e0aae1bb8c03fbad6864e30559a67ca"} Nov 02 14:36:31.788460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.788431 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz"] Nov 02 14:36:31.789155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789130 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1f2875bf-aabf-4099-b0be-390554272184" containerName="revoke-keys" Nov 02 14:36:31.789278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789156 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2875bf-aabf-4099-b0be-390554272184" containerName="revoke-keys" Nov 02 14:36:31.789278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789213 4431 cpu_manager.go:414] "containerMap: removing container" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" Nov 02 14:36:31.789278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789222 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" Nov 02 14:36:31.789278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789249 4431 cpu_manager.go:414] "containerMap: removing container" podUID="01e48e3a-374f-45e1-aac5-f26c9b89c55c" containerName="revoke-keys" Nov 02 14:36:31.789278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789257 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e48e3a-374f-45e1-aac5-f26c9b89c55c" containerName="revoke-keys" Nov 02 14:36:31.789278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789269 4431 cpu_manager.go:414] "containerMap: removing container" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" Nov 02 14:36:31.789278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789279 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" Nov 02 14:36:31.789641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789438 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" Nov 02 14:36:31.789641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789456 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e48e3a-374f-45e1-aac5-f26c9b89c55c" containerName="revoke-keys" Nov 02 14:36:31.789641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789466 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d580e7-f40b-47ea-b707-e1c6c3e144ac" containerName="istio-proxy" Nov 02 14:36:31.789641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.789480 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2875bf-aabf-4099-b0be-390554272184" containerName="revoke-keys" Nov 02 14:36:31.795847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.795828 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:31.809111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.807838 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz"] Nov 02 14:36:31.821043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.821013 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:36:31.821328 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.821306 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:36:31.827742 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.827722 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:36:31.861370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.861348 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:36:31.891431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.891412 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:36:31.903198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.903171 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:36:31.954188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954151 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-socket\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954197 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-istio-token\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-istio-token\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954244 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-kube-api-access-vd7d7\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-kube-api-access-vd7d7\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954338 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-data\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954367 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5152eb70-3989-4a4d-a6e7-5df967b974ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152eb70-3989-4a4d-a6e7-5df967b974ac-istiod-ca-cert\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954404 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-credential-socket\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954410 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-socket" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954456 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-credential-socket\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954486 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-podinfo\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954521 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-certs\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954604 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-socket\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.954735 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-credential-socket" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955089 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-envoy\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955140 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-data\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955380 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-credential-socket" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955407 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0881c94-9060-442e-b479-17013ddeb3ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0881c94-9060-442e-b479-17013ddeb3ca-istio-podinfo\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955460 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-certs\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955508 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-kube-api-access-js8pp\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-kube-api-access-js8pp\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955736 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-token\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-token\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955807 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0881c94-9060-442e-b479-17013ddeb3ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0881c94-9060-442e-b479-17013ddeb3ca-istiod-ca-cert\") pod \"e0881c94-9060-442e-b479-17013ddeb3ca\" (UID: \"e0881c94-9060-442e-b479-17013ddeb3ca\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.955857 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-envoy\") pod \"5152eb70-3989-4a4d-a6e7-5df967b974ac\" (UID: \"5152eb70-3989-4a4d-a6e7-5df967b974ac\") " Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.956268 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwhqc\" (UniqueName: \"kubernetes.io/projected/a0d52c50-0249-423b-a18c-870855ddca19-kube-api-access-zwhqc\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.956356 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.956446 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.956722 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.956741 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.956756 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.957009 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-data" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.957117 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-podinfo" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.957425 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0881c94-9060-442e-b479-17013ddeb3ca-istiod-ca-cert" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:31.958902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.957607 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-certs" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.960671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.959316 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0881c94-9060-442e-b479-17013ddeb3ca-istio-podinfo" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:36:31.960671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.959599 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-kube-api-access-js8pp" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "kube-api-access-js8pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:31.960671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.960107 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5152eb70-3989-4a4d-a6e7-5df967b974ac-istiod-ca-cert" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:31.960671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.960447 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-certs" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.961738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.961708 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-token" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:31.962017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.961914 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-envoy" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.962880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.962841 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-socket" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.963012 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.962951 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-data" pod "5152eb70-3989-4a4d-a6e7-5df967b974ac" (UID: "5152eb70-3989-4a4d-a6e7-5df967b974ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:31.966793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.963239 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-istio-token" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:31.966793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.963929 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-kube-api-access-vd7d7" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "kube-api-access-vd7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:31.966793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:31.965900 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-envoy" pod "e0881c94-9060-442e-b479-17013ddeb3ca" (UID: "e0881c94-9060-442e-b479-17013ddeb3ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.057687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.057593 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1927399f-2905-4061-8966-8a5d2d98cbda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1927399f-2905-4061-8966-8a5d2d98cbda-istiod-ca-cert\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.057687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.057643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-token\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-token\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.057687 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.057680 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-data\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.058084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.057719 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-socket\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.058084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.057750 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f2d68a-4f50-4fec-afed-5a20268f857c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f2d68a-4f50-4fec-afed-5a20268f857c-istiod-ca-cert\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.058084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.057866 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-podinfo\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.058084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.057945 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-istio-token\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-istio-token\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.058084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.057978 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-certs\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.058084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058023 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-kube-api-access-wd686\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-kube-api-access-wd686\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.058084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058051 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1927399f-2905-4061-8966-8a5d2d98cbda-istiod-ca-cert" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:32.058084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058072 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-certs\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.058562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058320 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-certs" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.058562 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058385 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-certs" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058797 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f2d68a-4f50-4fec-afed-5a20268f857c-istiod-ca-cert" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058887 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-credential-socket\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058948 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-data\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.058991 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-socket\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059029 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-envoy\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059062 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-envoy\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059096 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-credential-socket\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059117 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1927399f-2905-4061-8966-8a5d2d98cbda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1927399f-2905-4061-8966-8a5d2d98cbda-istio-podinfo\") pod \"1927399f-2905-4061-8966-8a5d2d98cbda\" (UID: \"1927399f-2905-4061-8966-8a5d2d98cbda\") " Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059190 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-kube-api-access-mw8wz\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-kube-api-access-mw8wz\") pod \"09f2d68a-4f50-4fec-afed-5a20268f857c\" (UID: \"09f2d68a-4f50-4fec-afed-5a20268f857c\") " Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059387 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwhqc\" (UniqueName: \"kubernetes.io/projected/a0d52c50-0249-423b-a18c-870855ddca19-kube-api-access-zwhqc\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059497 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-data" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059696 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-credential-socket" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.059860 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-credential-socket" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.060327 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-socket" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.060461 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-data" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.060513 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-socket" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.060891 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.060940 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.060959 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-podinfo" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061320 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061340 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061351 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0881c94-9060-442e-b479-17013ddeb3ca-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061362 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061355 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-istio-token" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061374 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8pp\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-kube-api-access-js8pp\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061388 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061399 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061411 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061421 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061431 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061442 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0881c94-9060-442e-b479-17013ddeb3ca-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061453 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061466 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061476 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061488 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7d7\" (UniqueName: \"kubernetes.io/projected/e0881c94-9060-442e-b479-17013ddeb3ca-kube-api-access-vd7d7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061500 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1927399f-2905-4061-8966-8a5d2d98cbda-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061512 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061447 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-kube-api-access-wd686" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "kube-api-access-wd686". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061527 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061560 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f2d68a-4f50-4fec-afed-5a20268f857c-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061577 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061591 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152eb70-3989-4a4d-a6e7-5df967b974ac-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061605 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152eb70-3989-4a4d-a6e7-5df967b974ac-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061618 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061631 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152eb70-3989-4a4d-a6e7-5df967b974ac-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061645 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061658 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0881c94-9060-442e-b479-17013ddeb3ca-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.061759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.061711 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:32.064402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.063373 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:32.064588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.064561 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-envoy" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.064826 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.064764 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-envoy" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:32.064942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.064905 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1927399f-2905-4061-8966-8a5d2d98cbda-istio-podinfo" pod "1927399f-2905-4061-8966-8a5d2d98cbda" (UID: "1927399f-2905-4061-8966-8a5d2d98cbda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:36:32.066040 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.066016 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-token" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:32.066545 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.066514 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-kube-api-access-mw8wz" pod "09f2d68a-4f50-4fec-afed-5a20268f857c" (UID: "09f2d68a-4f50-4fec-afed-5a20268f857c"). InnerVolumeSpecName "kube-api-access-mw8wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:32.086642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.086619 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwhqc\" (UniqueName: \"kubernetes.io/projected/a0d52c50-0249-423b-a18c-870855ddca19-kube-api-access-zwhqc\") pod \"maas-api-revoke-sub-f8b73c02-px6qz\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:32.160201 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.160172 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:32.162864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.162837 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.162864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.162859 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1927399f-2905-4061-8966-8a5d2d98cbda-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.162864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.162868 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1927399f-2905-4061-8966-8a5d2d98cbda-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.163165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.162877 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw8wz\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-kube-api-access-mw8wz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.163165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.162886 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f2d68a-4f50-4fec-afed-5a20268f857c-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.163165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.162894 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.163165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.162902 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd686\" (UniqueName: \"kubernetes.io/projected/1927399f-2905-4061-8966-8a5d2d98cbda-kube-api-access-wd686\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:32.308593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.308516 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd"] Nov 02 14:36:32.489444 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.489415 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz"] Nov 02 14:36:32.491753 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:36:32.491727 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d52c50_0249_423b_a18c_870855ddca19.slice/crio-82440674d5e1f5e2d8f06d009a3646dffd0f7b4c2869a050ee5ae96026665fc7 WatchSource:0}: Error finding container 82440674d5e1f5e2d8f06d009a3646dffd0f7b4c2869a050ee5ae96026665fc7: Status 404 returned error can't find the container with id 82440674d5e1f5e2d8f06d009a3646dffd0f7b4c2869a050ee5ae96026665fc7 Nov 02 14:36:32.659942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.659910 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" event={"ID":"a0d52c50-0249-423b-a18c-870855ddca19","Type":"ContainerStarted","Data":"82440674d5e1f5e2d8f06d009a3646dffd0f7b4c2869a050ee5ae96026665fc7"} Nov 02 14:36:32.661293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.661271 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" Nov 02 14:36:32.661417 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.661291 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5" event={"ID":"09f2d68a-4f50-4fec-afed-5a20268f857c","Type":"ContainerDied","Data":"3ac8d775bb3675d86c06b3da8c66869307f599c0dcca84a278e63df2e50833ea"} Nov 02 14:36:32.661417 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.661355 4431 scope.go:122] "RemoveContainer" containerID="01f1a807ee3f63eacc4ff59eac55f49a6cec4662c9d8106df8d6f194d0f4ff37" Nov 02 14:36:32.662854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.662822 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" Nov 02 14:36:32.662854 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.662840 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc" event={"ID":"1927399f-2905-4061-8966-8a5d2d98cbda","Type":"ContainerDied","Data":"ac6b21282a778fb0a310cc1d48badd537d83e3699b5135870b7b3fbc3c957c23"} Nov 02 14:36:32.664454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.664435 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" Nov 02 14:36:32.664454 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.664441 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l" event={"ID":"e0881c94-9060-442e-b479-17013ddeb3ca","Type":"ContainerDied","Data":"f494ad6fccc0d3c8a05db67e4cebd660d17b549652c5550ece0ca92090c22a28"} Nov 02 14:36:32.666182 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.666136 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" Nov 02 14:36:32.666182 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.666167 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx" event={"ID":"5152eb70-3989-4a4d-a6e7-5df967b974ac","Type":"ContainerDied","Data":"69fde2699503798d1f4ab60bc88b8d7007a3c950a62e8739cdb023ca3811e7a2"} Nov 02 14:36:32.666491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.666452 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" containerID="cri-o://f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181" gracePeriod=30 Nov 02 14:36:32.674299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.674255 4431 scope.go:122] "RemoveContainer" containerID="d1e3a8568c79cff14bb00cbd2d2c2c243f785f218e9f97ef861111dfe8d280ab" Nov 02 14:36:32.693760 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.693740 4431 scope.go:122] "RemoveContainer" containerID="a41e333a41a2841c798878e3432d12c4619efe4934e743a23c5e49b9ca2f5e41" Nov 02 14:36:32.702167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.702144 4431 scope.go:122] "RemoveContainer" containerID="e3dd0ce0de42a37ed1ecd91115c486fb9e0aae1bb8c03fbad6864e30559a67ca" Nov 02 14:36:32.734988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.734968 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5"] Nov 02 14:36:32.739257 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.739238 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-cknl5"] Nov 02 14:36:32.891107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.891084 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx"] Nov 02 14:36:32.904111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.904090 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-97tgx"] Nov 02 14:36:32.942154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.942102 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l"] Nov 02 14:36:32.946638 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.946614 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-jbt2l"] Nov 02 14:36:32.984713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.984691 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc"] Nov 02 14:36:32.989558 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:32.989533 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-pxflc"] Nov 02 14:36:33.076213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.076184 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" path="/var/lib/kubelet/pods/09f2d68a-4f50-4fec-afed-5a20268f857c/volumes" Nov 02 14:36:33.076620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.076607 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" path="/var/lib/kubelet/pods/1927399f-2905-4061-8966-8a5d2d98cbda/volumes" Nov 02 14:36:33.077051 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.077037 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" path="/var/lib/kubelet/pods/5152eb70-3989-4a4d-a6e7-5df967b974ac/volumes" Nov 02 14:36:33.077755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.077741 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" path="/var/lib/kubelet/pods/e0881c94-9060-442e-b479-17013ddeb3ca/volumes" Nov 02 14:36:33.327112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.327022 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb"] Nov 02 14:36:33.327813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.327771 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" Nov 02 14:36:33.328022 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.327996 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" Nov 02 14:36:33.328156 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328051 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" Nov 02 14:36:33.328156 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328060 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" Nov 02 14:36:33.328156 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328129 4431 cpu_manager.go:414] "containerMap: removing container" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" Nov 02 14:36:33.328156 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328137 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" Nov 02 14:36:33.328397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328159 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" Nov 02 14:36:33.328397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328167 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" Nov 02 14:36:33.328397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328328 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1927399f-2905-4061-8966-8a5d2d98cbda" containerName="istio-proxy" Nov 02 14:36:33.328397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328349 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f2d68a-4f50-4fec-afed-5a20268f857c" containerName="istio-proxy" Nov 02 14:36:33.328397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328361 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5152eb70-3989-4a4d-a6e7-5df967b974ac" containerName="istio-proxy" Nov 02 14:36:33.328397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.328375 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0881c94-9060-442e-b479-17013ddeb3ca" containerName="istio-proxy" Nov 02 14:36:33.333424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.333404 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.353356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.353330 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb"] Nov 02 14:36:33.480036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.480009 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.480186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.480064 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfm4z\" (UniqueName: \"kubernetes.io/projected/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-kube-api-access-tfm4z\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.480291 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.480275 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.581024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.580961 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.581024 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.581012 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfm4z\" (UniqueName: \"kubernetes.io/projected/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-kube-api-access-tfm4z\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.581189 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.581103 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.581654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.581632 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.583323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.583300 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.606158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.606132 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfm4z\" (UniqueName: \"kubernetes.io/projected/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-kube-api-access-tfm4z\") pod \"maas-api-revoke-sub-951f9f12-thdlb\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.647301 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.647277 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:33.675071 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.675042 4431 generic.go:356] "Generic (PLEG): container finished" podID="a0d52c50-0249-423b-a18c-870855ddca19" containerID="e387ab51dad42fb4a98bac450bdc8461fe40b48319a4d1147a6fa5afa55f5dea" exitCode=0 Nov 02 14:36:33.675390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.675121 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" event={"ID":"a0d52c50-0249-423b-a18c-870855ddca19","Type":"ContainerDied","Data":"e387ab51dad42fb4a98bac450bdc8461fe40b48319a4d1147a6fa5afa55f5dea"} Nov 02 14:36:33.952357 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:33.952322 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb"] Nov 02 14:36:33.953611 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:36:33.953583 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc54af7_a3a8_4d2d_9033_a7a6da6f58cd.slice/crio-ea30ad9b41ed176d1266ea38993d93b73046e2742a53ffa923f9c7581a80e876 WatchSource:0}: Error finding container ea30ad9b41ed176d1266ea38993d93b73046e2742a53ffa923f9c7581a80e876: Status 404 returned error can't find the container with id ea30ad9b41ed176d1266ea38993d93b73046e2742a53ffa923f9c7581a80e876 Nov 02 14:36:34.687397 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:34.687367 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" event={"ID":"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd","Type":"ContainerStarted","Data":"47ca70dbc6a7026ba891e2e94d0110fa7738a577a98be9f60c30257871d8b9d1"} Nov 02 14:36:34.687822 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:34.687418 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" event={"ID":"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd","Type":"ContainerStarted","Data":"ea30ad9b41ed176d1266ea38993d93b73046e2742a53ffa923f9c7581a80e876"} Nov 02 14:36:34.762929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:34.762878 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" podStartSLOduration=1.762859831 podStartE2EDuration="1.762859831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:34.761195935 +0000 UTC m=+2359.798300870" watchObservedRunningTime="2026-11-02 14:36:34.762859831 +0000 UTC m=+2359.799964766" Nov 02 14:36:34.827338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:34.827317 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:35.000690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.000627 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d52c50-0249-423b-a18c-870855ddca19-kube-api-access-zwhqc\" (UniqueName: \"kubernetes.io/projected/a0d52c50-0249-423b-a18c-870855ddca19-kube-api-access-zwhqc\") pod \"a0d52c50-0249-423b-a18c-870855ddca19\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " Nov 02 14:36:35.000848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.000832 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-openshift-ca\") pod \"a0d52c50-0249-423b-a18c-870855ddca19\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " Nov 02 14:36:35.000904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.000895 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-xks-ca\") pod \"a0d52c50-0249-423b-a18c-870855ddca19\" (UID: \"a0d52c50-0249-423b-a18c-870855ddca19\") " Nov 02 14:36:35.001190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.001163 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-openshift-ca" pod "a0d52c50-0249-423b-a18c-870855ddca19" (UID: "a0d52c50-0249-423b-a18c-870855ddca19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:35.002615 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.002590 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d52c50-0249-423b-a18c-870855ddca19-kube-api-access-zwhqc" pod "a0d52c50-0249-423b-a18c-870855ddca19" (UID: "a0d52c50-0249-423b-a18c-870855ddca19"). InnerVolumeSpecName "kube-api-access-zwhqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:35.002719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.002692 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-xks-ca" pod "a0d52c50-0249-423b-a18c-870855ddca19" (UID: "a0d52c50-0249-423b-a18c-870855ddca19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:35.101966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.101941 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:35.101966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.101964 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwhqc\" (UniqueName: \"kubernetes.io/projected/a0d52c50-0249-423b-a18c-870855ddca19-kube-api-access-zwhqc\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:35.102099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.101974 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d52c50-0249-423b-a18c-870855ddca19-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:35.702964 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.702932 4431 generic.go:356] "Generic (PLEG): container finished" podID="fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd" containerID="47ca70dbc6a7026ba891e2e94d0110fa7738a577a98be9f60c30257871d8b9d1" exitCode=0 Nov 02 14:36:35.703413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.703021 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" event={"ID":"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd","Type":"ContainerDied","Data":"47ca70dbc6a7026ba891e2e94d0110fa7738a577a98be9f60c30257871d8b9d1"} Nov 02 14:36:35.704617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.704590 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" event={"ID":"a0d52c50-0249-423b-a18c-870855ddca19","Type":"ContainerDied","Data":"82440674d5e1f5e2d8f06d009a3646dffd0f7b4c2869a050ee5ae96026665fc7"} Nov 02 14:36:35.704617 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.704613 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz" Nov 02 14:36:35.704843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:35.704632 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82440674d5e1f5e2d8f06d009a3646dffd0f7b4c2869a050ee5ae96026665fc7" Nov 02 14:36:36.294035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:36.294004 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc"] Nov 02 14:36:36.301111 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:36.301085 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk"] Nov 02 14:36:36.857969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:36.857948 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:37.021161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.021086 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-openshift-ca\") pod \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " Nov 02 14:36:37.021329 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.021241 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-xks-ca\") pod \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " Nov 02 14:36:37.021406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.021358 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-kube-api-access-tfm4z\" (UniqueName: \"kubernetes.io/projected/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-kube-api-access-tfm4z\") pod \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\" (UID: \"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd\") " Nov 02 14:36:37.021445 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.021411 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-openshift-ca" pod "fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd" (UID: "fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:37.021729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.021712 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:37.023325 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.023297 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-xks-ca" pod "fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd" (UID: "fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:37.023409 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.023340 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-kube-api-access-tfm4z" pod "fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd" (UID: "fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd"). InnerVolumeSpecName "kube-api-access-tfm4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:37.122650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.122622 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfm4z\" (UniqueName: \"kubernetes.io/projected/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-kube-api-access-tfm4z\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:37.122650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.122648 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:37.562921 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.562885 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn"] Nov 02 14:36:37.563350 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.563338 4431 cpu_manager.go:414] "containerMap: removing container" podUID="a0d52c50-0249-423b-a18c-870855ddca19" containerName="revoke-keys" Nov 02 14:36:37.563406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.563351 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d52c50-0249-423b-a18c-870855ddca19" containerName="revoke-keys" Nov 02 14:36:37.563406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.563381 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd" containerName="revoke-keys" Nov 02 14:36:37.563406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.563386 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd" containerName="revoke-keys" Nov 02 14:36:37.563505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.563487 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d52c50-0249-423b-a18c-870855ddca19" containerName="revoke-keys" Nov 02 14:36:37.563505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.563499 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd" containerName="revoke-keys" Nov 02 14:36:37.567213 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.567195 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.594247 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.594224 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn"] Nov 02 14:36:37.719731 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.719707 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" Nov 02 14:36:37.719885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.719751 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb" event={"ID":"fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd","Type":"ContainerDied","Data":"ea30ad9b41ed176d1266ea38993d93b73046e2742a53ffa923f9c7581a80e876"} Nov 02 14:36:37.719885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.719832 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea30ad9b41ed176d1266ea38993d93b73046e2742a53ffa923f9c7581a80e876" Nov 02 14:36:37.729548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.729524 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67wn\" (UniqueName: \"kubernetes.io/projected/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-kube-api-access-p67wn\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.729655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.729571 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.729655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.729597 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.830829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.830750 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67wn\" (UniqueName: \"kubernetes.io/projected/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-kube-api-access-p67wn\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.830829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.830807 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.831015 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.830845 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.831541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.831511 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.833474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.833448 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.863751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.863723 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67wn\" (UniqueName: \"kubernetes.io/projected/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-kube-api-access-p67wn\") pod \"maas-api-revoke-sub-213987ab-bddsn\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.880444 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.880417 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:37.951812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:37.941563 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:36:38.134524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134450 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-socket\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.134524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134512 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-podinfo\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.134759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134554 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-envoy\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.134759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134615 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-token\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-token\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.134759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-kube-api-access-w2sk6\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-kube-api-access-w2sk6\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.134759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134716 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-socket" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:38.134759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134744 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-data\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.135557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134812 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istiod-ca-cert\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.135557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134865 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-certs\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.135557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.134958 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-credential-socket\") pod \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\" (UID: \"8aad9b7c-e120-4cb6-82dd-14a81f078af9\") " Nov 02 14:36:38.135557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.135033 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-data" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:38.135557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.135240 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-certs" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:38.135557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.135274 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istiod-ca-cert" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:38.135557 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.135453 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-credential-socket" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:38.135908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.135559 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.135908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.135578 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.135908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.135593 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.135908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.135607 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.137127 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.137103 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-token" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:38.137127 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.137120 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-envoy" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:38.137312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.137138 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-podinfo" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:36:38.137713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.137694 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-kube-api-access-w2sk6" pod "8aad9b7c-e120-4cb6-82dd-14a81f078af9" (UID: "8aad9b7c-e120-4cb6-82dd-14a81f078af9"). InnerVolumeSpecName "kube-api-access-w2sk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:38.185841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.185822 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn"] Nov 02 14:36:38.186547 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:36:38.186517 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6eb92ed_6be3_4f1b_b4ae_ef761ad3b26e.slice/crio-9efc92e9332191c734c346da2f1866a2dd358a7e6fd9ba50f462a0e2489f8531 WatchSource:0}: Error finding container 9efc92e9332191c734c346da2f1866a2dd358a7e6fd9ba50f462a0e2489f8531: Status 404 returned error can't find the container with id 9efc92e9332191c734c346da2f1866a2dd358a7e6fd9ba50f462a0e2489f8531 Nov 02 14:36:38.237225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.237203 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.237225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.237225 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.237407 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.237236 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.237407 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.237244 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.237407 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.237251 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2sk6\" (UniqueName: \"kubernetes.io/projected/8aad9b7c-e120-4cb6-82dd-14a81f078af9-kube-api-access-w2sk6\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:38.729460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.729425 4431 generic.go:356] "Generic (PLEG): container finished" podID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerID="f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181" exitCode=0 Nov 02 14:36:38.729632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.729493 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" Nov 02 14:36:38.729632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.729509 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" event={"ID":"8aad9b7c-e120-4cb6-82dd-14a81f078af9","Type":"ContainerDied","Data":"f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181"} Nov 02 14:36:38.729632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.729579 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" event={"ID":"8aad9b7c-e120-4cb6-82dd-14a81f078af9","Type":"ContainerDied","Data":"ec2398500c5be77fdb38340675f8019ddcbe6d6bf66c4cdd43fe3e9feab26345"} Nov 02 14:36:38.729632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.729610 4431 scope.go:122] "RemoveContainer" containerID="f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181" Nov 02 14:36:38.731278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.731251 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" event={"ID":"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e","Type":"ContainerStarted","Data":"b5837bc788ae9b37255a754b45660809cecd3bda23a128d6aaa1b04000cbbd67"} Nov 02 14:36:38.731388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.731295 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" event={"ID":"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e","Type":"ContainerStarted","Data":"9efc92e9332191c734c346da2f1866a2dd358a7e6fd9ba50f462a0e2489f8531"} Nov 02 14:36:38.740599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.740500 4431 scope.go:122] "RemoveContainer" containerID="a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019" Nov 02 14:36:38.750064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.750043 4431 scope.go:122] "RemoveContainer" containerID="f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181" Nov 02 14:36:38.750354 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:36:38.750336 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181\": container with ID starting with f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181 not found: ID does not exist" containerID="f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181" Nov 02 14:36:38.750424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.750364 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181"} err="failed to get container status \"f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181\": rpc error: code = NotFound desc = could not find container \"f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181\": container with ID starting with f2878b2c13ef353382cf4e716969e236b8204b3e8e8a2711fb41c098da9ef181 not found: ID does not exist" Nov 02 14:36:38.750424 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.750390 4431 scope.go:122] "RemoveContainer" containerID="a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019" Nov 02 14:36:38.750639 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:36:38.750623 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019\": container with ID starting with a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019 not found: ID does not exist" containerID="a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019" Nov 02 14:36:38.750701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.750647 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019"} err="failed to get container status \"a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019\": rpc error: code = NotFound desc = could not find container \"a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019\": container with ID starting with a3cc38615db2b4dc83240165dfe533028513b7bbfae848e616a97fbcd513a019 not found: ID does not exist" Nov 02 14:36:38.802061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.802033 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd"] Nov 02 14:36:38.814369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.814343 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd"] Nov 02 14:36:38.845092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.845054 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" podStartSLOduration=1.845042389 podStartE2EDuration="1.845042389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:38.843212101 +0000 UTC m=+2363.880317034" watchObservedRunningTime="2026-11-02 14:36:38.845042389 +0000 UTC m=+2363.882147321" Nov 02 14:36:38.895664 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.895636 4431 patch_prober.go:29] interesting pod/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:36:38.896021 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:38.895681 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-90a96161-openshift-default-85fd94cfcc-gblmd" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:36:39.076062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:39.076020 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" path="/var/lib/kubelet/pods/8aad9b7c-e120-4cb6-82dd-14a81f078af9/volumes" Nov 02 14:36:39.739872 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:39.739838 4431 generic.go:356] "Generic (PLEG): container finished" podID="b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e" containerID="b5837bc788ae9b37255a754b45660809cecd3bda23a128d6aaa1b04000cbbd67" exitCode=0 Nov 02 14:36:39.740064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:39.739926 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" event={"ID":"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e","Type":"ContainerDied","Data":"b5837bc788ae9b37255a754b45660809cecd3bda23a128d6aaa1b04000cbbd67"} Nov 02 14:36:40.893064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:40.893040 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:40.961330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:40.961301 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-kube-api-access-p67wn\" (UniqueName: \"kubernetes.io/projected/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-kube-api-access-p67wn\") pod \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " Nov 02 14:36:40.961491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:40.961343 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-xks-ca\") pod \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " Nov 02 14:36:40.961491 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:40.961392 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-openshift-ca\") pod \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\" (UID: \"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e\") " Nov 02 14:36:40.961838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:40.961805 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-openshift-ca" pod "b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e" (UID: "b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:40.963641 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:40.963613 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-xks-ca" pod "b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e" (UID: "b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:40.963898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:40.963869 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-kube-api-access-p67wn" pod "b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e" (UID: "b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e"). InnerVolumeSpecName "kube-api-access-p67wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:41.062453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:41.062387 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67wn\" (UniqueName: \"kubernetes.io/projected/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-kube-api-access-p67wn\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:41.062453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:41.062423 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:41.062453 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:41.062433 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:41.753438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:41.753411 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" Nov 02 14:36:41.753622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:41.753453 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn" event={"ID":"b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e","Type":"ContainerDied","Data":"9efc92e9332191c734c346da2f1866a2dd358a7e6fd9ba50f462a0e2489f8531"} Nov 02 14:36:41.753622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:41.753498 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9efc92e9332191c734c346da2f1866a2dd358a7e6fd9ba50f462a0e2489f8531" Nov 02 14:36:44.305749 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.305715 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6"] Nov 02 14:36:44.306266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.306251 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" Nov 02 14:36:44.306309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.306267 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" Nov 02 14:36:44.306309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.306288 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e" containerName="revoke-keys" Nov 02 14:36:44.306309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.306294 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e" containerName="revoke-keys" Nov 02 14:36:44.306411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.306379 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" Nov 02 14:36:44.306411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.306389 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" Nov 02 14:36:44.306411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.306396 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e" containerName="revoke-keys" Nov 02 14:36:44.309735 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.309721 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.332144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.332103 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:36:44.345187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.345168 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:36:44.350359 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.350335 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6"] Nov 02 14:36:44.398227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.398198 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqk8\" (UniqueName: \"kubernetes.io/projected/d2a2a004-da83-4244-8f22-53d3eef3c29d-kube-api-access-pwqk8\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.398351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.398232 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.398351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.398267 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.499850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.499817 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqk8\" (UniqueName: \"kubernetes.io/projected/d2a2a004-da83-4244-8f22-53d3eef3c29d-kube-api-access-pwqk8\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.499969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.499855 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.499969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.499890 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.500546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.500527 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.502099 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.502083 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.525229 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.525205 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqk8\" (UniqueName: \"kubernetes.io/projected/d2a2a004-da83-4244-8f22-53d3eef3c29d-kube-api-access-pwqk8\") pod \"maas-api-revoke-sub-cc9f6d62-tdmb6\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.622586 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.622530 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:44.973631 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:44.973605 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6"] Nov 02 14:36:44.975197 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:36:44.975157 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a2a004_da83_4244_8f22_53d3eef3c29d.slice/crio-efceaaeceffe98b1b45ead6702cdef5e355fc635b0fd5f60a575c68c35d6e87f WatchSource:0}: Error finding container efceaaeceffe98b1b45ead6702cdef5e355fc635b0fd5f60a575c68c35d6e87f: Status 404 returned error can't find the container with id efceaaeceffe98b1b45ead6702cdef5e355fc635b0fd5f60a575c68c35d6e87f Nov 02 14:36:45.781711 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:45.781677 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" event={"ID":"d2a2a004-da83-4244-8f22-53d3eef3c29d","Type":"ContainerStarted","Data":"012abd83c66fa9a94f23947929b4c77b5437c7fee16b248342450e7c21f98b75"} Nov 02 14:36:45.782104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:45.781739 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" event={"ID":"d2a2a004-da83-4244-8f22-53d3eef3c29d","Type":"ContainerStarted","Data":"efceaaeceffe98b1b45ead6702cdef5e355fc635b0fd5f60a575c68c35d6e87f"} Nov 02 14:36:45.836415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:45.836367 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" podStartSLOduration=1.836353093 podStartE2EDuration="1.836353093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:45.833946085 +0000 UTC m=+2370.871051018" watchObservedRunningTime="2026-11-02 14:36:45.836353093 +0000 UTC m=+2370.873458024" Nov 02 14:36:46.790109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:46.790072 4431 generic.go:356] "Generic (PLEG): container finished" podID="d2a2a004-da83-4244-8f22-53d3eef3c29d" containerID="012abd83c66fa9a94f23947929b4c77b5437c7fee16b248342450e7c21f98b75" exitCode=0 Nov 02 14:36:46.790550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:46.790151 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" event={"ID":"d2a2a004-da83-4244-8f22-53d3eef3c29d","Type":"ContainerDied","Data":"012abd83c66fa9a94f23947929b4c77b5437c7fee16b248342450e7c21f98b75"} Nov 02 14:36:47.945340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:47.945317 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:48.073647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.073574 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a2a004-da83-4244-8f22-53d3eef3c29d-kube-api-access-pwqk8\" (UniqueName: \"kubernetes.io/projected/d2a2a004-da83-4244-8f22-53d3eef3c29d-kube-api-access-pwqk8\") pod \"d2a2a004-da83-4244-8f22-53d3eef3c29d\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " Nov 02 14:36:48.073820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.073664 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-openshift-ca\") pod \"d2a2a004-da83-4244-8f22-53d3eef3c29d\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " Nov 02 14:36:48.073820 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.073698 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-xks-ca\") pod \"d2a2a004-da83-4244-8f22-53d3eef3c29d\" (UID: \"d2a2a004-da83-4244-8f22-53d3eef3c29d\") " Nov 02 14:36:48.074072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.074051 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-openshift-ca" pod "d2a2a004-da83-4244-8f22-53d3eef3c29d" (UID: "d2a2a004-da83-4244-8f22-53d3eef3c29d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:36:48.075541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.075513 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a2a004-da83-4244-8f22-53d3eef3c29d-kube-api-access-pwqk8" pod "d2a2a004-da83-4244-8f22-53d3eef3c29d" (UID: "d2a2a004-da83-4244-8f22-53d3eef3c29d"). InnerVolumeSpecName "kube-api-access-pwqk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:48.075613 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.075556 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-xks-ca" pod "d2a2a004-da83-4244-8f22-53d3eef3c29d" (UID: "d2a2a004-da83-4244-8f22-53d3eef3c29d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:48.175083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.175057 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:48.175083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.175084 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwqk8\" (UniqueName: \"kubernetes.io/projected/d2a2a004-da83-4244-8f22-53d3eef3c29d-kube-api-access-pwqk8\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:48.175295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.175098 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a2a004-da83-4244-8f22-53d3eef3c29d-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:48.806413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.806382 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" Nov 02 14:36:48.806621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.806383 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6" event={"ID":"d2a2a004-da83-4244-8f22-53d3eef3c29d","Type":"ContainerDied","Data":"efceaaeceffe98b1b45ead6702cdef5e355fc635b0fd5f60a575c68c35d6e87f"} Nov 02 14:36:48.806621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:48.806502 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efceaaeceffe98b1b45ead6702cdef5e355fc635b0fd5f60a575c68c35d6e87f" Nov 02 14:36:51.283901 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:36:51.283861 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:36:51.316272 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.316243 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" containerID="cri-o://e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636" gracePeriod=45 Nov 02 14:36:51.327152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.327120 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" containerID="cri-o://75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965" gracePeriod=45 Nov 02 14:36:51.364882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.364856 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": dial tcp 10.133.0.86:8000: connect: connection refused" Nov 02 14:36:51.601721 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.601697 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:36:51.605044 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.605024 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:36:51.711679 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.711639 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47e4c06-0b1c-464b-b0e7-768229c9cc35-kube-api-access-tw4xj\" (UniqueName: \"kubernetes.io/projected/b47e4c06-0b1c-464b-b0e7-768229c9cc35-kube-api-access-tw4xj\") pod \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " Nov 02 14:36:51.711897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.711690 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-home\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-home\") pod \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " Nov 02 14:36:51.711897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.711741 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tmp-dir\") pod \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " Nov 02 14:36:51.711897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.711788 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tls-certs\" (UniqueName: \"kubernetes.io/secret/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tls-certs\") pod \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " Nov 02 14:36:51.711897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.711841 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tls-certs\") pod \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " Nov 02 14:36:51.712163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.711929 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-model-cache\") pod \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " Nov 02 14:36:51.712163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.711993 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e2ef36-f412-4e56-a735-8a035e9fddbc-kube-api-access-kn6zr\" (UniqueName: \"kubernetes.io/projected/e9e2ef36-f412-4e56-a735-8a035e9fddbc-kube-api-access-kn6zr\") pod \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " Nov 02 14:36:51.712163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712039 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-dshm\") pod \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " Nov 02 14:36:51.712163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712052 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-home" pod "e9e2ef36-f412-4e56-a735-8a035e9fddbc" (UID: "e9e2ef36-f412-4e56-a735-8a035e9fddbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.712163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712074 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-home\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-home\") pod \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " Nov 02 14:36:51.712163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712101 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tmp-dir\") pod \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\" (UID: \"b47e4c06-0b1c-464b-b0e7-768229c9cc35\") " Nov 02 14:36:51.712163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712144 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-dshm\") pod \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " Nov 02 14:36:51.712548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712234 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-model-cache\") pod \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\" (UID: \"e9e2ef36-f412-4e56-a735-8a035e9fddbc\") " Nov 02 14:36:51.712548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712394 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tmp-dir" pod "e9e2ef36-f412-4e56-a735-8a035e9fddbc" (UID: "e9e2ef36-f412-4e56-a735-8a035e9fddbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.712662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712579 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-home" pod "b47e4c06-0b1c-464b-b0e7-768229c9cc35" (UID: "b47e4c06-0b1c-464b-b0e7-768229c9cc35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.712750 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712729 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.712851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712755 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.712851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.712768 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.713332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.713100 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-model-cache" pod "b47e4c06-0b1c-464b-b0e7-768229c9cc35" (UID: "b47e4c06-0b1c-464b-b0e7-768229c9cc35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.713332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.713312 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tmp-dir" pod "b47e4c06-0b1c-464b-b0e7-768229c9cc35" (UID: "b47e4c06-0b1c-464b-b0e7-768229c9cc35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.713332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.713323 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-model-cache" pod "e9e2ef36-f412-4e56-a735-8a035e9fddbc" (UID: "e9e2ef36-f412-4e56-a735-8a035e9fddbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.715181 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.715155 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e2ef36-f412-4e56-a735-8a035e9fddbc-kube-api-access-kn6zr" pod "e9e2ef36-f412-4e56-a735-8a035e9fddbc" (UID: "e9e2ef36-f412-4e56-a735-8a035e9fddbc"). InnerVolumeSpecName "kube-api-access-kn6zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:51.715701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.715676 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tls-certs" pod "b47e4c06-0b1c-464b-b0e7-768229c9cc35" (UID: "b47e4c06-0b1c-464b-b0e7-768229c9cc35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:51.715841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.715808 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47e4c06-0b1c-464b-b0e7-768229c9cc35-kube-api-access-tw4xj" pod "b47e4c06-0b1c-464b-b0e7-768229c9cc35" (UID: "b47e4c06-0b1c-464b-b0e7-768229c9cc35"). InnerVolumeSpecName "kube-api-access-tw4xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:36:51.716455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.716425 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-dshm" pod "e9e2ef36-f412-4e56-a735-8a035e9fddbc" (UID: "e9e2ef36-f412-4e56-a735-8a035e9fddbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.716971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.716931 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tls-certs" pod "e9e2ef36-f412-4e56-a735-8a035e9fddbc" (UID: "e9e2ef36-f412-4e56-a735-8a035e9fddbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:36:51.718376 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.718349 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-dshm" pod "b47e4c06-0b1c-464b-b0e7-768229c9cc35" (UID: "b47e4c06-0b1c-464b-b0e7-768229c9cc35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:36:51.814418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814384 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e2ef36-f412-4e56-a735-8a035e9fddbc-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.814418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814418 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.814627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814432 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn6zr\" (UniqueName: \"kubernetes.io/projected/e9e2ef36-f412-4e56-a735-8a035e9fddbc-kube-api-access-kn6zr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.814627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814444 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.814627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814455 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.814627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814466 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.814627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814476 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e2ef36-f412-4e56-a735-8a035e9fddbc-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.814627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814488 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw4xj\" (UniqueName: \"kubernetes.io/projected/b47e4c06-0b1c-464b-b0e7-768229c9cc35-kube-api-access-tw4xj\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.814627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.814499 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47e4c06-0b1c-464b-b0e7-768229c9cc35-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:36:51.826146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.826072 4431 generic.go:356] "Generic (PLEG): container finished" podID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerID="e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636" exitCode=0 Nov 02 14:36:51.826146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.826120 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" Nov 02 14:36:51.826342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.826153 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" event={"ID":"e9e2ef36-f412-4e56-a735-8a035e9fddbc","Type":"ContainerDied","Data":"e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636"} Nov 02 14:36:51.826342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.826191 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc" event={"ID":"e9e2ef36-f412-4e56-a735-8a035e9fddbc","Type":"ContainerDied","Data":"7d228d89a6c74e945993639d0e319db24e92dc630d5123e71b7f0fd588b0e814"} Nov 02 14:36:51.826342 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.826212 4431 scope.go:122] "RemoveContainer" containerID="e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636" Nov 02 14:36:51.827703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.827676 4431 generic.go:356] "Generic (PLEG): container finished" podID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerID="75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965" exitCode=0 Nov 02 14:36:51.827845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.827711 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" Nov 02 14:36:51.827845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.827762 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" event={"ID":"b47e4c06-0b1c-464b-b0e7-768229c9cc35","Type":"ContainerDied","Data":"75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965"} Nov 02 14:36:51.827845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.827825 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk" event={"ID":"b47e4c06-0b1c-464b-b0e7-768229c9cc35","Type":"ContainerDied","Data":"e52958cce927a465b557fc907b4e9f9e05a8d79f3f9773bcfe2de583a6c37826"} Nov 02 14:36:51.838296 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.838276 4431 scope.go:122] "RemoveContainer" containerID="e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636" Nov 02 14:36:51.838587 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:36:51.838565 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636\": container with ID starting with e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636 not found: ID does not exist" containerID="e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636" Nov 02 14:36:51.838657 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.838594 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636"} err="failed to get container status \"e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636\": rpc error: code = NotFound desc = could not find container \"e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636\": container with ID starting with e1fb2765e2caf500881b8d8bc7c0c54ecc05e0dd63e47600441d5ec820428636 not found: ID does not exist" Nov 02 14:36:51.838657 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.838612 4431 scope.go:122] "RemoveContainer" containerID="75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965" Nov 02 14:36:51.851370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.851343 4431 scope.go:122] "RemoveContainer" containerID="75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965" Nov 02 14:36:51.851722 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:36:51.851703 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965\": container with ID starting with 75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965 not found: ID does not exist" containerID="75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965" Nov 02 14:36:51.851811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.851732 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965"} err="failed to get container status \"75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965\": rpc error: code = NotFound desc = could not find container \"75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965\": container with ID starting with 75398533cdef575f5de3c51e030f7cd975b691c38da172a66b0c485066b60965 not found: ID does not exist" Nov 02 14:36:51.884023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.884001 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk"] Nov 02 14:36:51.886829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.886811 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/premium-w2-4be645-kserve-8d7b895db-5jjmk"] Nov 02 14:36:51.921837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.921812 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc"] Nov 02 14:36:51.924374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:51.924351 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4be645/distinct-w2-4be645-kserve-85c98cc649-rq7nc"] Nov 02 14:36:53.076333 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:53.076299 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" path="/var/lib/kubelet/pods/b47e4c06-0b1c-464b-b0e7-768229c9cc35/volumes" Nov 02 14:36:53.076718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:53.076705 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" path="/var/lib/kubelet/pods/e9e2ef36-f412-4e56-a735-8a035e9fddbc/volumes" Nov 02 14:36:54.870554 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:36:54.870522 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:37:11.708654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.708620 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp"] Nov 02 14:37:11.709315 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709294 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d2a2a004-da83-4244-8f22-53d3eef3c29d" containerName="revoke-keys" Nov 02 14:37:11.709430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709316 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a2a004-da83-4244-8f22-53d3eef3c29d" containerName="revoke-keys" Nov 02 14:37:11.709430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709334 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" Nov 02 14:37:11.709430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709341 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" Nov 02 14:37:11.709430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709366 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" Nov 02 14:37:11.709430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709374 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" Nov 02 14:37:11.709430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709403 4431 cpu_manager.go:414] "containerMap: removing container" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" Nov 02 14:37:11.709430 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709410 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aad9b7c-e120-4cb6-82dd-14a81f078af9" containerName="istio-proxy" Nov 02 14:37:11.709808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709555 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a2a004-da83-4244-8f22-53d3eef3c29d" containerName="revoke-keys" Nov 02 14:37:11.709808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709570 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47e4c06-0b1c-464b-b0e7-768229c9cc35" containerName="main" Nov 02 14:37:11.709808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.709581 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e2ef36-f412-4e56-a735-8a035e9fddbc" containerName="main" Nov 02 14:37:11.714905 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.714884 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.751268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.747666 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4tfr8\"" Nov 02 14:37:11.753063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.752905 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79"] Nov 02 14:37:11.759514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.759487 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q"] Nov 02 14:37:11.761541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.761519 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.765084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.765052 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp"] Nov 02 14:37:11.765202 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.765158 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.766033 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.766015 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79"] Nov 02 14:37:11.770330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.770309 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q"] Nov 02 14:37:11.815535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815514 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.815640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815545 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.815640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815572 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.815640 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815626 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.815838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815716 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc27444d-3855-4031-8b67-8fe90fb4ae96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.815838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815750 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.815838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815804 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.815973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815858 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.815973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815877 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.815973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815905 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22zt\" (UniqueName: \"kubernetes.io/projected/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-kube-api-access-g22zt\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.815973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815936 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.816124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.815976 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.816124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816043 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.816124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816064 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xthm\" (UniqueName: \"kubernetes.io/projected/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-kube-api-access-2xthm\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.816286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816131 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.816286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816150 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.816286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816167 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.816286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816215 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.816286 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816262 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.816451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816299 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.816451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816323 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.816451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816371 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.816451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816397 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.816451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816421 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.816611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816465 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.816611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816520 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.816611 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.816540 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznpx\" (UniqueName: \"kubernetes.io/projected/bc27444d-3855-4031-8b67-8fe90fb4ae96-kube-api-access-lznpx\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.917967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.917941 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.918121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.917971 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.917998 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc27444d-3855-4031-8b67-8fe90fb4ae96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.918121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918021 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.918313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918193 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918244 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918313 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918275 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.918474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918316 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22zt\" (UniqueName: \"kubernetes.io/projected/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-kube-api-access-g22zt\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.918474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918363 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918455 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.918647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918453 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918481 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.918647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918512 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.918647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918570 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918605 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xthm\" (UniqueName: \"kubernetes.io/projected/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-kube-api-access-2xthm\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918643 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918676 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.918970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918837 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.918970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918881 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.918970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918908 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.918970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918944 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918974 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.918998 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919026 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919034 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919052 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919016 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc27444d-3855-4031-8b67-8fe90fb4ae96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919090 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919119 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919170 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919196 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznpx\" (UniqueName: \"kubernetes.io/projected/bc27444d-3855-4031-8b67-8fe90fb4ae96-kube-api-access-lznpx\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919249 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-data\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919258 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.919308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919306 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.920057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919339 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.920057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919542 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.920057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.919594 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.920057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.920020 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-workload-certs\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.920299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.920135 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.920299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.920141 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.920423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.920385 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-workload-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.920547 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.920505 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-credential-socket\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.920975 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.920957 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.921245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.921225 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.921722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.921698 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.921981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.921963 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.922661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.922640 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-envoy\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.922874 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.922856 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.944366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.944334 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.945142 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.945123 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznpx\" (UniqueName: \"kubernetes.io/projected/bc27444d-3855-4031-8b67-8fe90fb4ae96-kube-api-access-lznpx\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.945242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.945150 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xthm\" (UniqueName: \"kubernetes.io/projected/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-kube-api-access-2xthm\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:11.946057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.946035 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22zt\" (UniqueName: \"kubernetes.io/projected/fefae9cc-8fdb-4c2a-9029-5218b3f2ea97-kube-api-access-g22zt\") pod \"maas-default-gateway-openshift-default-55fbd65575-t9gjp\" (UID: \"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:11.946256 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.946235 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc27444d-3855-4031-8b67-8fe90fb4ae96-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-q8r79\" (UID: \"bc27444d-3855-4031-8b67-8fe90fb4ae96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:11.952339 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:11.952316 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f771e0c6-2f81-4e2f-afd3-20be3bc4eddf-istio-token\") pod \"maas-default-gateway-openshift-default-55fbd65575-2m48q\" (UID: \"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:12.038152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.038087 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:12.091355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.091308 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:12.097924 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.097294 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:12.356541 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.356507 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp"] Nov 02 14:37:12.357077 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:37:12.357058 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefae9cc_8fdb_4c2a_9029_5218b3f2ea97.slice/crio-a9d85922cb7447def50ec054f85708789e7e5986a323dadcd28315635055d013 WatchSource:0}: Error finding container a9d85922cb7447def50ec054f85708789e7e5986a323dadcd28315635055d013: Status 404 returned error can't find the container with id a9d85922cb7447def50ec054f85708789e7e5986a323dadcd28315635055d013 Nov 02 14:37:12.359526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.359491 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.359637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.359571 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.359637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.359606 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.419271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.419244 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q"] Nov 02 14:37:12.420190 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:37:12.420170 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf771e0c6_2f81_4e2f_afd3_20be3bc4eddf.slice/crio-4964a6ab48e46e391a359e027919c5d8efecf8bff1d02ce7390a95be1b82c7cb WatchSource:0}: Error finding container 4964a6ab48e46e391a359e027919c5d8efecf8bff1d02ce7390a95be1b82c7cb: Status 404 returned error can't find the container with id 4964a6ab48e46e391a359e027919c5d8efecf8bff1d02ce7390a95be1b82c7cb Nov 02 14:37:12.422697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.422667 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.422800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.422735 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.422856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.422805 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.437368 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.437342 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79"] Nov 02 14:37:12.441494 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:37:12.441414 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc27444d_3855_4031_8b67_8fe90fb4ae96.slice/crio-1b78b332a1617e96725f7f6223ae1ca7e7d3e5568511d3f978263bc6064ebb91 WatchSource:0}: Error finding container 1b78b332a1617e96725f7f6223ae1ca7e7d3e5568511d3f978263bc6064ebb91: Status 404 returned error can't find the container with id 1b78b332a1617e96725f7f6223ae1ca7e7d3e5568511d3f978263bc6064ebb91 Nov 02 14:37:12.444109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.444070 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.444208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.444136 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.444208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.444179 4431 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 14:37:12.978587 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.978550 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" event={"ID":"bc27444d-3855-4031-8b67-8fe90fb4ae96","Type":"ContainerStarted","Data":"1de466cccfffc7ba31b0e7cccd472029b8ccc22b16918c1a9ef6d0eeec123138"} Nov 02 14:37:12.979186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.979162 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" event={"ID":"bc27444d-3855-4031-8b67-8fe90fb4ae96","Type":"ContainerStarted","Data":"1b78b332a1617e96725f7f6223ae1ca7e7d3e5568511d3f978263bc6064ebb91"} Nov 02 14:37:12.982919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.982892 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" event={"ID":"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf","Type":"ContainerStarted","Data":"c182fbedd705dd2ec6c93c36146cf291f9f7421982b05434dd3ad02943bef742"} Nov 02 14:37:12.983117 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:12.983102 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" event={"ID":"f771e0c6-2f81-4e2f-afd3-20be3bc4eddf","Type":"ContainerStarted","Data":"4964a6ab48e46e391a359e027919c5d8efecf8bff1d02ce7390a95be1b82c7cb"} Nov 02 14:37:13.106116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.101695 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:37:13.106116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.101759 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:37:13.114270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.110016 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 14:37:13.114270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.110084 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 14:37:13.119086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.119028 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podStartSLOduration=2.118992187 podStartE2EDuration="2.118992187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:13.096280218 +0000 UTC m=+2398.133385151" watchObservedRunningTime="2026-11-02 14:37:13.118992187 +0000 UTC m=+2398.156097119" Nov 02 14:37:13.216838 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.210497 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podStartSLOduration=2.210479236 podStartE2EDuration="2.210479236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:13.209129241 +0000 UTC m=+2398.246234173" watchObservedRunningTime="2026-11-02 14:37:13.210479236 +0000 UTC m=+2398.247584166" Nov 02 14:37:13.254437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.254354 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:13.254715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.254690 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" event={"ID":"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97","Type":"ContainerStarted","Data":"36b536488ed019118051c5900b96015c5c7dedebe52ca118b6bb7f10c21fde12"} Nov 02 14:37:13.255433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.255414 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:13.255563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.255548 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" event={"ID":"fefae9cc-8fdb-4c2a-9029-5218b3f2ea97","Type":"ContainerStarted","Data":"a9d85922cb7447def50ec054f85708789e7e5986a323dadcd28315635055d013"} Nov 02 14:37:13.270929 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:13.270846 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podStartSLOduration=2.270826462 podStartE2EDuration="2.270826462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:13.270393856 +0000 UTC m=+2398.307498793" watchObservedRunningTime="2026-11-02 14:37:13.270826462 +0000 UTC m=+2398.307931393" Nov 02 14:37:14.043231 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:14.043143 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:14.056829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:14.048378 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 14:37:14.056829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:14.048435 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 14:37:14.100563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:14.100526 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 14:37:14.100853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:14.100822 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 14:37:14.101614 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:14.101590 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:37:14.101811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:14.101771 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:37:15.049465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:15.045505 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 14:37:15.049465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:15.045576 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 14:37:15.091890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:15.091850 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:37:15.092103 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:15.091915 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:37:15.111799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:15.107388 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 14:37:15.113179 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:15.113147 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 14:37:16.041942 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:16.041908 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 14:37:16.042706 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:16.042674 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 14:37:16.092498 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:16.092463 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:37:16.097127 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:16.097095 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:37:16.126800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:16.122493 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 14:37:16.126800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:16.122548 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 14:37:17.041711 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:17.041225 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 14:37:17.041711 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:17.041292 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 14:37:17.093620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:17.093507 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:37:17.093620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:17.093564 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:37:17.108937 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:17.108903 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 14:37:17.109215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:17.109187 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 14:37:18.055507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:18.053950 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 14:37:18.055507 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:18.054014 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 14:37:18.101550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:18.101116 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:37:18.101550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:18.101174 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:37:18.102116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:18.101768 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 14:37:18.102116 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:18.101841 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 14:37:18.127253 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:18.127221 4431 scope.go:122] "RemoveContainer" containerID="08b457406a97ff27851ba41748184bfb918cfb56f2d25f06b5f35818a2846b4d" Nov 02 14:37:18.167902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:18.167877 4431 scope.go:122] "RemoveContainer" containerID="dc787fb315bc65b0f6f98b00ac2ea8f0ff0270b8e299695e0eed593194e959a5" Nov 02 14:37:19.038692 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:19.038620 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 14:37:19.038898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:19.038699 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 14:37:19.092019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:19.091986 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:37:19.092236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:19.092050 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:37:19.098456 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:19.098423 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 14:37:19.098607 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:19.098478 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 14:37:20.038864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:20.038829 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body= Nov 02 14:37:20.039355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:20.038906 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" Nov 02 14:37:20.091806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:20.091765 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 02 14:37:20.091955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:20.091829 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 02 14:37:20.098243 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:20.098216 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body= Nov 02 14:37:20.098365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:20.098265 4431 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" Nov 02 14:37:21.428202 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:37:21.428151 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:37:21.832287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.832183 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:21.833047 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.833022 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:21.833809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.833766 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" Nov 02 14:37:21.885598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.885560 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:21.886842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.886817 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:21.887544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.887517 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" Nov 02 14:37:21.946450 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.946384 4431 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:21.946871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.946828 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:21.948014 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:21.947990 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" Nov 02 14:37:51.287860 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:37:51.283822 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:37:54.152522 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:54.152489 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:37:57.634396 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.634359 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj"] Nov 02 14:37:57.639568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.639549 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.659821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.659800 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:37:57.673961 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.673940 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:37:57.677458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.677433 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj"] Nov 02 14:37:57.743351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.743325 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.743472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.743359 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45f4\" (UniqueName: \"kubernetes.io/projected/e03993b8-0205-4fab-ad05-6a86d755dc0d-kube-api-access-n45f4\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.743472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.743439 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.845057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.845031 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.845174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.845066 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45f4\" (UniqueName: \"kubernetes.io/projected/e03993b8-0205-4fab-ad05-6a86d755dc0d-kube-api-access-n45f4\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.845221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.845191 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.845693 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.845661 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.847431 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.847408 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.877806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.877787 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45f4\" (UniqueName: \"kubernetes.io/projected/e03993b8-0205-4fab-ad05-6a86d755dc0d-kube-api-access-n45f4\") pod \"maas-api-revoke-sub-222b8b47-26lvj\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:57.952622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:57.952591 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:37:58.265373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:58.265347 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj"] Nov 02 14:37:58.266352 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:37:58.266310 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03993b8_0205_4fab_ad05_6a86d755dc0d.slice/crio-f1e1a35290df3e27f503195c6b9591363e63e6fe54b03d8ee84e715e1fc69918 WatchSource:0}: Error finding container f1e1a35290df3e27f503195c6b9591363e63e6fe54b03d8ee84e715e1fc69918: Status 404 returned error can't find the container with id f1e1a35290df3e27f503195c6b9591363e63e6fe54b03d8ee84e715e1fc69918 Nov 02 14:37:58.623680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:58.623652 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" event={"ID":"e03993b8-0205-4fab-ad05-6a86d755dc0d","Type":"ContainerStarted","Data":"ede9fdde13581f64b5719219b383c44e33de2b929b054e15c1ce3bbd7cf676a4"} Nov 02 14:37:58.623847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:58.623708 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" event={"ID":"e03993b8-0205-4fab-ad05-6a86d755dc0d","Type":"ContainerStarted","Data":"f1e1a35290df3e27f503195c6b9591363e63e6fe54b03d8ee84e715e1fc69918"} Nov 02 14:37:58.670627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:58.670586 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" podStartSLOduration=1.670571816 podStartE2EDuration="1.670571816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:58.667668413 +0000 UTC m=+2443.704773355" watchObservedRunningTime="2026-11-02 14:37:58.670571816 +0000 UTC m=+2443.707676748" Nov 02 14:37:59.632277 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:59.632175 4431 generic.go:356] "Generic (PLEG): container finished" podID="e03993b8-0205-4fab-ad05-6a86d755dc0d" containerID="ede9fdde13581f64b5719219b383c44e33de2b929b054e15c1ce3bbd7cf676a4" exitCode=0 Nov 02 14:37:59.632451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:37:59.632269 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" event={"ID":"e03993b8-0205-4fab-ad05-6a86d755dc0d","Type":"ContainerDied","Data":"ede9fdde13581f64b5719219b383c44e33de2b929b054e15c1ce3bbd7cf676a4"} Nov 02 14:38:00.796295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.796270 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:38:00.878792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.878759 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-openshift-ca\") pod \"e03993b8-0205-4fab-ad05-6a86d755dc0d\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " Nov 02 14:38:00.878971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.878877 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03993b8-0205-4fab-ad05-6a86d755dc0d-kube-api-access-n45f4\" (UniqueName: \"kubernetes.io/projected/e03993b8-0205-4fab-ad05-6a86d755dc0d-kube-api-access-n45f4\") pod \"e03993b8-0205-4fab-ad05-6a86d755dc0d\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " Nov 02 14:38:00.878971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.878940 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-xks-ca\") pod \"e03993b8-0205-4fab-ad05-6a86d755dc0d\" (UID: \"e03993b8-0205-4fab-ad05-6a86d755dc0d\") " Nov 02 14:38:00.879204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.879178 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-openshift-ca" pod "e03993b8-0205-4fab-ad05-6a86d755dc0d" (UID: "e03993b8-0205-4fab-ad05-6a86d755dc0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:00.879392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.879369 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:00.881068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.881044 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03993b8-0205-4fab-ad05-6a86d755dc0d-kube-api-access-n45f4" pod "e03993b8-0205-4fab-ad05-6a86d755dc0d" (UID: "e03993b8-0205-4fab-ad05-6a86d755dc0d"). InnerVolumeSpecName "kube-api-access-n45f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:00.881068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.881060 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-xks-ca" pod "e03993b8-0205-4fab-ad05-6a86d755dc0d" (UID: "e03993b8-0205-4fab-ad05-6a86d755dc0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:38:00.980650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.980618 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n45f4\" (UniqueName: \"kubernetes.io/projected/e03993b8-0205-4fab-ad05-6a86d755dc0d-kube-api-access-n45f4\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:00.980650 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:00.980646 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03993b8-0205-4fab-ad05-6a86d755dc0d-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:01.646121 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:01.646081 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" event={"ID":"e03993b8-0205-4fab-ad05-6a86d755dc0d","Type":"ContainerDied","Data":"f1e1a35290df3e27f503195c6b9591363e63e6fe54b03d8ee84e715e1fc69918"} Nov 02 14:38:01.646305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:01.646160 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e1a35290df3e27f503195c6b9591363e63e6fe54b03d8ee84e715e1fc69918" Nov 02 14:38:01.646305 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:01.646105 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj" Nov 02 14:38:08.045392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.045359 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2"] Nov 02 14:38:08.045957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.045939 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e03993b8-0205-4fab-ad05-6a86d755dc0d" containerName="revoke-keys" Nov 02 14:38:08.046017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.045958 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03993b8-0205-4fab-ad05-6a86d755dc0d" containerName="revoke-keys" Nov 02 14:38:08.046096 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.046086 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03993b8-0205-4fab-ad05-6a86d755dc0d" containerName="revoke-keys" Nov 02 14:38:08.049606 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.049591 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.070472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.070442 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2"] Nov 02 14:38:08.079796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.079769 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:38:08.079932 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.079772 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:38:08.167889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.167851 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.168057 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.167908 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fnp4\" (UniqueName: \"kubernetes.io/projected/1fee25df-4efc-4825-892e-5edb2df791f4-kube-api-access-5fnp4\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.168140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.168050 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.268772 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.268752 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.268866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.268824 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fnp4\" (UniqueName: \"kubernetes.io/projected/1fee25df-4efc-4825-892e-5edb2df791f4-kube-api-access-5fnp4\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.268906 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.268866 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.269439 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.269415 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.271038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.271018 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.300139 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.300068 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fnp4\" (UniqueName: \"kubernetes.io/projected/1fee25df-4efc-4825-892e-5edb2df791f4-kube-api-access-5fnp4\") pod \"maas-api-revoke-sub-04864e63-8qjl2\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.362981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.362946 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:08.674979 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.674949 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2"] Nov 02 14:38:08.675756 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:38:08.675730 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fee25df_4efc_4825_892e_5edb2df791f4.slice/crio-be261c193e809d8531a20e1740b50beb1c6e18ce10d99b327aecfe56ebec6140 WatchSource:0}: Error finding container be261c193e809d8531a20e1740b50beb1c6e18ce10d99b327aecfe56ebec6140: Status 404 returned error can't find the container with id be261c193e809d8531a20e1740b50beb1c6e18ce10d99b327aecfe56ebec6140 Nov 02 14:38:08.689927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:08.689896 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" event={"ID":"1fee25df-4efc-4825-892e-5edb2df791f4","Type":"ContainerStarted","Data":"be261c193e809d8531a20e1740b50beb1c6e18ce10d99b327aecfe56ebec6140"} Nov 02 14:38:09.698372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:09.698340 4431 generic.go:356] "Generic (PLEG): container finished" podID="1fee25df-4efc-4825-892e-5edb2df791f4" containerID="5ba9594be1158df329810d1eb76543d453ae8221a7371842eab14b8a9e687604" exitCode=0 Nov 02 14:38:09.698737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:09.698383 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" event={"ID":"1fee25df-4efc-4825-892e-5edb2df791f4","Type":"ContainerDied","Data":"5ba9594be1158df329810d1eb76543d453ae8221a7371842eab14b8a9e687604"} Nov 02 14:38:10.853040 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:10.853015 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:10.997804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:10.997696 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-openshift-ca\") pod \"1fee25df-4efc-4825-892e-5edb2df791f4\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " Nov 02 14:38:10.997956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:10.997828 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-xks-ca\") pod \"1fee25df-4efc-4825-892e-5edb2df791f4\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " Nov 02 14:38:10.997956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:10.997943 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fee25df-4efc-4825-892e-5edb2df791f4-kube-api-access-5fnp4\" (UniqueName: \"kubernetes.io/projected/1fee25df-4efc-4825-892e-5edb2df791f4-kube-api-access-5fnp4\") pod \"1fee25df-4efc-4825-892e-5edb2df791f4\" (UID: \"1fee25df-4efc-4825-892e-5edb2df791f4\") " Nov 02 14:38:10.998080 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:10.998057 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-openshift-ca" pod "1fee25df-4efc-4825-892e-5edb2df791f4" (UID: "1fee25df-4efc-4825-892e-5edb2df791f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:10.998323 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:10.998300 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:11.000088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:11.000062 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-xks-ca" pod "1fee25df-4efc-4825-892e-5edb2df791f4" (UID: "1fee25df-4efc-4825-892e-5edb2df791f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:38:11.000197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:11.000174 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fee25df-4efc-4825-892e-5edb2df791f4-kube-api-access-5fnp4" pod "1fee25df-4efc-4825-892e-5edb2df791f4" (UID: "1fee25df-4efc-4825-892e-5edb2df791f4"). InnerVolumeSpecName "kube-api-access-5fnp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:11.099882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:11.099853 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fee25df-4efc-4825-892e-5edb2df791f4-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:11.099882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:11.099879 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fnp4\" (UniqueName: \"kubernetes.io/projected/1fee25df-4efc-4825-892e-5edb2df791f4-kube-api-access-5fnp4\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:11.712598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:11.712571 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" Nov 02 14:38:11.712819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:11.712571 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2" event={"ID":"1fee25df-4efc-4825-892e-5edb2df791f4","Type":"ContainerDied","Data":"be261c193e809d8531a20e1740b50beb1c6e18ce10d99b327aecfe56ebec6140"} Nov 02 14:38:11.712819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:11.712692 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be261c193e809d8531a20e1740b50beb1c6e18ce10d99b327aecfe56ebec6140" Nov 02 14:38:18.879240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.879208 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67"] Nov 02 14:38:18.879729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.879715 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1fee25df-4efc-4825-892e-5edb2df791f4" containerName="revoke-keys" Nov 02 14:38:18.879794 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.879730 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fee25df-4efc-4825-892e-5edb2df791f4" containerName="revoke-keys" Nov 02 14:38:18.879870 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.879859 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fee25df-4efc-4825-892e-5edb2df791f4" containerName="revoke-keys" Nov 02 14:38:18.883038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.883024 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:18.898127 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.898102 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:18.898241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.898142 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:18.898293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.898271 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8pxj\" (UniqueName: \"kubernetes.io/projected/abfb67e0-8791-4275-8c11-04d8d03d2a32-kube-api-access-v8pxj\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:18.905593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.905545 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67"] Nov 02 14:38:18.914395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.914375 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:38:18.914487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:18.914408 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:38:19.000042 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.000019 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:19.000167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.000051 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:19.000167 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.000106 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8pxj\" (UniqueName: \"kubernetes.io/projected/abfb67e0-8791-4275-8c11-04d8d03d2a32-kube-api-access-v8pxj\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:19.000667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.000645 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:19.002358 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.002341 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:19.026755 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.026728 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8pxj\" (UniqueName: \"kubernetes.io/projected/abfb67e0-8791-4275-8c11-04d8d03d2a32-kube-api-access-v8pxj\") pod \"maas-api-revoke-sub-67194783-4ms67\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:19.043077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.043054 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q"] Nov 02 14:38:19.196797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.196412 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:19.534289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.534251 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67"] Nov 02 14:38:19.538991 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:38:19.538964 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfb67e0_8791_4275_8c11_04d8d03d2a32.slice/crio-700dbf92d2d58d475c82f3640ba03a3e98762315bfecdef53c5fe72a0222c07e WatchSource:0}: Error finding container 700dbf92d2d58d475c82f3640ba03a3e98762315bfecdef53c5fe72a0222c07e: Status 404 returned error can't find the container with id 700dbf92d2d58d475c82f3640ba03a3e98762315bfecdef53c5fe72a0222c07e Nov 02 14:38:19.762626 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:19.762596 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" event={"ID":"abfb67e0-8791-4275-8c11-04d8d03d2a32","Type":"ContainerStarted","Data":"700dbf92d2d58d475c82f3640ba03a3e98762315bfecdef53c5fe72a0222c07e"} Nov 02 14:38:20.771856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:20.771758 4431 generic.go:356] "Generic (PLEG): container finished" podID="abfb67e0-8791-4275-8c11-04d8d03d2a32" containerID="ee203d2c02d3144dea1daad2c897286215a45de4e054ea9ba4e5440172e8508f" exitCode=0 Nov 02 14:38:20.771856 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:20.771814 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" event={"ID":"abfb67e0-8791-4275-8c11-04d8d03d2a32","Type":"ContainerDied","Data":"ee203d2c02d3144dea1daad2c897286215a45de4e054ea9ba4e5440172e8508f"} Nov 02 14:38:21.283540 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:38:21.283507 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:38:21.930081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:21.930059 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:22.045523 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.045447 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfb67e0-8791-4275-8c11-04d8d03d2a32-kube-api-access-v8pxj\" (UniqueName: \"kubernetes.io/projected/abfb67e0-8791-4275-8c11-04d8d03d2a32-kube-api-access-v8pxj\") pod \"abfb67e0-8791-4275-8c11-04d8d03d2a32\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " Nov 02 14:38:22.045663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.045553 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-xks-ca\") pod \"abfb67e0-8791-4275-8c11-04d8d03d2a32\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " Nov 02 14:38:22.045663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.045629 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-openshift-ca\") pod \"abfb67e0-8791-4275-8c11-04d8d03d2a32\" (UID: \"abfb67e0-8791-4275-8c11-04d8d03d2a32\") " Nov 02 14:38:22.046019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.045989 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-openshift-ca" pod "abfb67e0-8791-4275-8c11-04d8d03d2a32" (UID: "abfb67e0-8791-4275-8c11-04d8d03d2a32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:22.047548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.047518 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfb67e0-8791-4275-8c11-04d8d03d2a32-kube-api-access-v8pxj" pod "abfb67e0-8791-4275-8c11-04d8d03d2a32" (UID: "abfb67e0-8791-4275-8c11-04d8d03d2a32"). InnerVolumeSpecName "kube-api-access-v8pxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:22.047645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.047562 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-xks-ca" pod "abfb67e0-8791-4275-8c11-04d8d03d2a32" (UID: "abfb67e0-8791-4275-8c11-04d8d03d2a32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:38:22.147039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.147001 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:22.147039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.147030 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb67e0-8791-4275-8c11-04d8d03d2a32-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:22.147226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.147046 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8pxj\" (UniqueName: \"kubernetes.io/projected/abfb67e0-8791-4275-8c11-04d8d03d2a32-kube-api-access-v8pxj\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:22.788422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.788390 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" Nov 02 14:38:22.788422 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.788402 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67" event={"ID":"abfb67e0-8791-4275-8c11-04d8d03d2a32","Type":"ContainerDied","Data":"700dbf92d2d58d475c82f3640ba03a3e98762315bfecdef53c5fe72a0222c07e"} Nov 02 14:38:22.788629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:22.788453 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="700dbf92d2d58d475c82f3640ba03a3e98762315bfecdef53c5fe72a0222c07e" Nov 02 14:38:33.039839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:33.039809 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582"] Nov 02 14:38:33.052671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:33.052647 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80b341aa-tn582"] Nov 02 14:38:33.075990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:33.075966 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4" path="/var/lib/kubelet/pods/0e9c60c2-8e85-41a5-b5a6-d0b6b4fa81a4/volumes" Nov 02 14:38:34.063443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.063385 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" podUID="6bb3534d-fc5b-4180-b613-6135efd7a12c" containerName="main" containerID="cri-o://479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652" gracePeriod=45 Nov 02 14:38:34.348842 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.348813 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:38:34.482207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482175 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-model-cache\") pod \"6bb3534d-fc5b-4180-b613-6135efd7a12c\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " Nov 02 14:38:34.482373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482279 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-tmp-dir\") pod \"6bb3534d-fc5b-4180-b613-6135efd7a12c\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " Nov 02 14:38:34.482373 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482346 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-home\") pod \"6bb3534d-fc5b-4180-b613-6135efd7a12c\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " Nov 02 14:38:34.482451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482417 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb3534d-fc5b-4180-b613-6135efd7a12c-kube-api-access-swmrh\" (UniqueName: \"kubernetes.io/projected/6bb3534d-fc5b-4180-b613-6135efd7a12c-kube-api-access-swmrh\") pod \"6bb3534d-fc5b-4180-b613-6135efd7a12c\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " Nov 02 14:38:34.482515 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482476 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-model-cache" pod "6bb3534d-fc5b-4180-b613-6135efd7a12c" (UID: "6bb3534d-fc5b-4180-b613-6135efd7a12c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:34.482561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482533 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-dshm\") pod \"6bb3534d-fc5b-4180-b613-6135efd7a12c\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " Nov 02 14:38:34.482597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482570 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb3534d-fc5b-4180-b613-6135efd7a12c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3534d-fc5b-4180-b613-6135efd7a12c-tls-certs\") pod \"6bb3534d-fc5b-4180-b613-6135efd7a12c\" (UID: \"6bb3534d-fc5b-4180-b613-6135efd7a12c\") " Nov 02 14:38:34.482597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482564 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-tmp-dir" pod "6bb3534d-fc5b-4180-b613-6135efd7a12c" (UID: "6bb3534d-fc5b-4180-b613-6135efd7a12c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:34.482662 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.482590 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-home" pod "6bb3534d-fc5b-4180-b613-6135efd7a12c" (UID: "6bb3534d-fc5b-4180-b613-6135efd7a12c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:34.483039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.483015 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:34.483039 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.483039 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:34.483266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.483053 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:34.484517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.484492 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-dshm" pod "6bb3534d-fc5b-4180-b613-6135efd7a12c" (UID: "6bb3534d-fc5b-4180-b613-6135efd7a12c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:38:34.484627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.484534 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb3534d-fc5b-4180-b613-6135efd7a12c-kube-api-access-swmrh" pod "6bb3534d-fc5b-4180-b613-6135efd7a12c" (UID: "6bb3534d-fc5b-4180-b613-6135efd7a12c"). InnerVolumeSpecName "kube-api-access-swmrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:34.484627 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.484584 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb3534d-fc5b-4180-b613-6135efd7a12c-tls-certs" pod "6bb3534d-fc5b-4180-b613-6135efd7a12c" (UID: "6bb3534d-fc5b-4180-b613-6135efd7a12c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:38:34.583708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.583643 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmrh\" (UniqueName: \"kubernetes.io/projected/6bb3534d-fc5b-4180-b613-6135efd7a12c-kube-api-access-swmrh\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:34.583708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.583670 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb3534d-fc5b-4180-b613-6135efd7a12c-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:34.583708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.583679 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb3534d-fc5b-4180-b613-6135efd7a12c-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:34.815140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.815108 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:38:34.866577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.866510 4431 generic.go:356] "Generic (PLEG): container finished" podID="6bb3534d-fc5b-4180-b613-6135efd7a12c" containerID="479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652" exitCode=0 Nov 02 14:38:34.866577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.866549 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" Nov 02 14:38:34.866744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.866589 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" event={"ID":"6bb3534d-fc5b-4180-b613-6135efd7a12c","Type":"ContainerDied","Data":"479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652"} Nov 02 14:38:34.866744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.866625 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q" event={"ID":"6bb3534d-fc5b-4180-b613-6135efd7a12c","Type":"ContainerDied","Data":"47d6ba696a34f77cea21f6398534dc5658f30eec108e7934fa0d15b2fb618507"} Nov 02 14:38:34.866744 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.866642 4431 scope.go:122] "RemoveContainer" containerID="479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652" Nov 02 14:38:34.876690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.876671 4431 scope.go:122] "RemoveContainer" containerID="479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652" Nov 02 14:38:34.876956 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:38:34.876936 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652\": container with ID starting with 479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652 not found: ID does not exist" containerID="479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652" Nov 02 14:38:34.877036 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.876966 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652"} err="failed to get container status \"479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652\": rpc error: code = NotFound desc = could not find container \"479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652\": container with ID starting with 479b99097d2cfcda395fdb3e0d37104aa379ff8bb371a9ade4856b0524d2c652 not found: ID does not exist" Nov 02 14:38:34.927497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.927472 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q"] Nov 02 14:38:34.930415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:34.930398 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e1c2/rate-test-model-f1e1c2-kserve-c85776749-75h6q"] Nov 02 14:38:35.075819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:35.075792 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb3534d-fc5b-4180-b613-6135efd7a12c" path="/var/lib/kubelet/pods/6bb3534d-fc5b-4180-b613-6135efd7a12c/volumes" Nov 02 14:38:44.455438 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.455403 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw"] Nov 02 14:38:44.456653 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.456627 4431 cpu_manager.go:414] "containerMap: removing container" podUID="6bb3534d-fc5b-4180-b613-6135efd7a12c" containerName="main" Nov 02 14:38:44.456653 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.456650 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb3534d-fc5b-4180-b613-6135efd7a12c" containerName="main" Nov 02 14:38:44.456844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.456668 4431 cpu_manager.go:414] "containerMap: removing container" podUID="abfb67e0-8791-4275-8c11-04d8d03d2a32" containerName="revoke-keys" Nov 02 14:38:44.456844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.456674 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfb67e0-8791-4275-8c11-04d8d03d2a32" containerName="revoke-keys" Nov 02 14:38:44.456844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.456807 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb3534d-fc5b-4180-b613-6135efd7a12c" containerName="main" Nov 02 14:38:44.456844 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.456818 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfb67e0-8791-4275-8c11-04d8d03d2a32" containerName="revoke-keys" Nov 02 14:38:44.460215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.460198 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.480811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.480772 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw"] Nov 02 14:38:44.489404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.489380 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:38:44.489404 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.489387 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:38:44.598293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.598262 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fqx\" (UniqueName: \"kubernetes.io/projected/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-kube-api-access-d7fqx\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.598448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.598344 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.598493 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.598474 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.699536 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.699503 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fqx\" (UniqueName: \"kubernetes.io/projected/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-kube-api-access-d7fqx\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.699706 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.699583 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.699757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.699712 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.700340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.700315 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.701864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.701837 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.726194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.726139 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fqx\" (UniqueName: \"kubernetes.io/projected/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-kube-api-access-d7fqx\") pod \"maas-api-revoke-sub-e35f0ffc-g6cfw\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.731569 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.731543 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t"] Nov 02 14:38:44.735715 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.735692 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:44.755846 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.755825 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t"] Nov 02 14:38:44.773190 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.773171 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:44.902126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.902087 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvm5\" (UniqueName: \"kubernetes.io/projected/eb998700-a763-44ad-9ce6-ce9d1007ea12-kube-api-access-sqvm5\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:44.902126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.902129 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:44.902363 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:44.902341 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:45.003866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.003800 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:45.003866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.003857 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvm5\" (UniqueName: \"kubernetes.io/projected/eb998700-a763-44ad-9ce6-ce9d1007ea12-kube-api-access-sqvm5\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:45.004053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.003977 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:45.004421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.004403 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:45.006241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.006219 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:45.028660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.028639 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvm5\" (UniqueName: \"kubernetes.io/projected/eb998700-a763-44ad-9ce6-ce9d1007ea12-kube-api-access-sqvm5\") pod \"maas-api-revoke-sub-32fce375-4gb4t\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:45.049629 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.049607 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:45.082567 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.082544 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw"] Nov 02 14:38:45.083569 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:38:45.083549 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6ac095_6aa1_46c9_a337_d8a08cd51fd7.slice/crio-8a0ec70d2d3991ba4c3112cf91cb153ab85af7f606dc74a859351c3909c41c0d WatchSource:0}: Error finding container 8a0ec70d2d3991ba4c3112cf91cb153ab85af7f606dc74a859351c3909c41c0d: Status 404 returned error can't find the container with id 8a0ec70d2d3991ba4c3112cf91cb153ab85af7f606dc74a859351c3909c41c0d Nov 02 14:38:45.358120 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.358095 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t"] Nov 02 14:38:45.359275 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:38:45.359248 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb998700_a763_44ad_9ce6_ce9d1007ea12.slice/crio-dbdeeb08c4786f82947fa5b0909396ba98ab805eabb9d8a7abc4929e05c2c58d WatchSource:0}: Error finding container dbdeeb08c4786f82947fa5b0909396ba98ab805eabb9d8a7abc4929e05c2c58d: Status 404 returned error can't find the container with id dbdeeb08c4786f82947fa5b0909396ba98ab805eabb9d8a7abc4929e05c2c58d Nov 02 14:38:45.940267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.940185 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" event={"ID":"eb998700-a763-44ad-9ce6-ce9d1007ea12","Type":"ContainerStarted","Data":"df2ae00422ffcae9a39a7777f40c3553683c9fc7355faa0bf45931420ffc20ca"} Nov 02 14:38:45.940267 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.940249 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" event={"ID":"eb998700-a763-44ad-9ce6-ce9d1007ea12","Type":"ContainerStarted","Data":"dbdeeb08c4786f82947fa5b0909396ba98ab805eabb9d8a7abc4929e05c2c58d"} Nov 02 14:38:45.942081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.942053 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" event={"ID":"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7","Type":"ContainerStarted","Data":"e430ec120ac9d00d043e64125cf1752fe1216ee7e5767c9acac93bb21c1ab1d1"} Nov 02 14:38:45.942211 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.942096 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" event={"ID":"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7","Type":"ContainerStarted","Data":"8a0ec70d2d3991ba4c3112cf91cb153ab85af7f606dc74a859351c3909c41c0d"} Nov 02 14:38:45.986427 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:45.986386 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" podStartSLOduration=1.986374045 podStartE2EDuration="1.986374045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:45.98293302 +0000 UTC m=+2491.020037951" watchObservedRunningTime="2026-11-02 14:38:45.986374045 +0000 UTC m=+2491.023478977" Nov 02 14:38:46.034294 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:46.034248 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" podStartSLOduration=2.034235049 podStartE2EDuration="2.034235049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:46.031150189 +0000 UTC m=+2491.068255121" watchObservedRunningTime="2026-11-02 14:38:46.034235049 +0000 UTC m=+2491.071339995" Nov 02 14:38:46.949819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:46.949767 4431 generic.go:356] "Generic (PLEG): container finished" podID="eb998700-a763-44ad-9ce6-ce9d1007ea12" containerID="df2ae00422ffcae9a39a7777f40c3553683c9fc7355faa0bf45931420ffc20ca" exitCode=0 Nov 02 14:38:46.949819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:46.949806 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" event={"ID":"eb998700-a763-44ad-9ce6-ce9d1007ea12","Type":"ContainerDied","Data":"df2ae00422ffcae9a39a7777f40c3553683c9fc7355faa0bf45931420ffc20ca"} Nov 02 14:38:46.951472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:46.951452 4431 generic.go:356] "Generic (PLEG): container finished" podID="3d6ac095-6aa1-46c9-a337-d8a08cd51fd7" containerID="e430ec120ac9d00d043e64125cf1752fe1216ee7e5767c9acac93bb21c1ab1d1" exitCode=0 Nov 02 14:38:46.951472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:46.951478 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" event={"ID":"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7","Type":"ContainerDied","Data":"e430ec120ac9d00d043e64125cf1752fe1216ee7e5767c9acac93bb21c1ab1d1"} Nov 02 14:38:48.114244 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.114217 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:48.144821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.144800 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:48.238050 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.237961 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-openshift-ca\") pod \"eb998700-a763-44ad-9ce6-ce9d1007ea12\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " Nov 02 14:38:48.238198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.238103 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-xks-ca\") pod \"eb998700-a763-44ad-9ce6-ce9d1007ea12\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " Nov 02 14:38:48.238198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.238152 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb998700-a763-44ad-9ce6-ce9d1007ea12-kube-api-access-sqvm5\" (UniqueName: \"kubernetes.io/projected/eb998700-a763-44ad-9ce6-ce9d1007ea12-kube-api-access-sqvm5\") pod \"eb998700-a763-44ad-9ce6-ce9d1007ea12\" (UID: \"eb998700-a763-44ad-9ce6-ce9d1007ea12\") " Nov 02 14:38:48.238411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.238388 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-openshift-ca" pod "eb998700-a763-44ad-9ce6-ce9d1007ea12" (UID: "eb998700-a763-44ad-9ce6-ce9d1007ea12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:48.238698 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.238676 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:48.240266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.240231 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb998700-a763-44ad-9ce6-ce9d1007ea12-kube-api-access-sqvm5" pod "eb998700-a763-44ad-9ce6-ce9d1007ea12" (UID: "eb998700-a763-44ad-9ce6-ce9d1007ea12"). InnerVolumeSpecName "kube-api-access-sqvm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:48.240374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.240304 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-xks-ca" pod "eb998700-a763-44ad-9ce6-ce9d1007ea12" (UID: "eb998700-a763-44ad-9ce6-ce9d1007ea12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:38:48.339710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.339682 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-openshift-ca\") pod \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " Nov 02 14:38:48.339868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.339844 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-kube-api-access-d7fqx\" (UniqueName: \"kubernetes.io/projected/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-kube-api-access-d7fqx\") pod \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " Nov 02 14:38:48.339952 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.339933 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-xks-ca\") pod \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\" (UID: \"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7\") " Nov 02 14:38:48.340094 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.340071 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-openshift-ca" pod "3d6ac095-6aa1-46c9-a337-d8a08cd51fd7" (UID: "3d6ac095-6aa1-46c9-a337-d8a08cd51fd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:48.340382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.340364 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:48.340463 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.340382 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb998700-a763-44ad-9ce6-ce9d1007ea12-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:48.340463 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.340393 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvm5\" (UniqueName: \"kubernetes.io/projected/eb998700-a763-44ad-9ce6-ce9d1007ea12-kube-api-access-sqvm5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:48.341920 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.341896 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-kube-api-access-d7fqx" pod "3d6ac095-6aa1-46c9-a337-d8a08cd51fd7" (UID: "3d6ac095-6aa1-46c9-a337-d8a08cd51fd7"). InnerVolumeSpecName "kube-api-access-d7fqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:48.342017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.341924 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-xks-ca" pod "3d6ac095-6aa1-46c9-a337-d8a08cd51fd7" (UID: "3d6ac095-6aa1-46c9-a337-d8a08cd51fd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:38:48.441707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.441683 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fqx\" (UniqueName: \"kubernetes.io/projected/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-kube-api-access-d7fqx\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:48.441707 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.441706 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:48.957839 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.957806 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q"] Nov 02 14:38:48.964666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.964644 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" Nov 02 14:38:48.964850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.964688 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t" event={"ID":"eb998700-a763-44ad-9ce6-ce9d1007ea12","Type":"ContainerDied","Data":"dbdeeb08c4786f82947fa5b0909396ba98ab805eabb9d8a7abc4929e05c2c58d"} Nov 02 14:38:48.964850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.964734 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbdeeb08c4786f82947fa5b0909396ba98ab805eabb9d8a7abc4929e05c2c58d" Nov 02 14:38:48.966335 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.966312 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" Nov 02 14:38:48.966465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.966369 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw" event={"ID":"3d6ac095-6aa1-46c9-a337-d8a08cd51fd7","Type":"ContainerDied","Data":"8a0ec70d2d3991ba4c3112cf91cb153ab85af7f606dc74a859351c3909c41c0d"} Nov 02 14:38:48.966465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:48.966408 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0ec70d2d3991ba4c3112cf91cb153ab85af7f606dc74a859351c3909c41c0d" Nov 02 14:38:51.282797 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:38:51.282744 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:38:55.075293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.075255 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f"] Nov 02 14:38:55.075939 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.075920 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3d6ac095-6aa1-46c9-a337-d8a08cd51fd7" containerName="revoke-keys" Nov 02 14:38:55.076062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.075940 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6ac095-6aa1-46c9-a337-d8a08cd51fd7" containerName="revoke-keys" Nov 02 14:38:55.076062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.075965 4431 cpu_manager.go:414] "containerMap: removing container" podUID="eb998700-a763-44ad-9ce6-ce9d1007ea12" containerName="revoke-keys" Nov 02 14:38:55.076062 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.075971 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb998700-a763-44ad-9ce6-ce9d1007ea12" containerName="revoke-keys" Nov 02 14:38:55.076232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.076142 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6ac095-6aa1-46c9-a337-d8a08cd51fd7" containerName="revoke-keys" Nov 02 14:38:55.076232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.076159 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb998700-a763-44ad-9ce6-ce9d1007ea12" containerName="revoke-keys" Nov 02 14:38:55.080802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.080769 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.093469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.093446 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f"] Nov 02 14:38:55.102748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.102727 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:38:55.102867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.102767 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:38:55.161797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.161745 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.161914 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.161895 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksz7\" (UniqueName: \"kubernetes.io/projected/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-kube-api-access-cksz7\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.161971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.161941 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.263441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.263422 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.263550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.263477 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksz7\" (UniqueName: \"kubernetes.io/projected/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-kube-api-access-cksz7\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.263598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.263583 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.264204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.264183 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.265743 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.265723 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.288140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.288117 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksz7\" (UniqueName: \"kubernetes.io/projected/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-kube-api-access-cksz7\") pod \"maas-api-revoke-sub-f4931a8c-n6m7f\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.394648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.394629 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:55.698394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:55.698364 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f"] Nov 02 14:38:55.698891 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:38:55.698867 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6f069c_113b_42a2_84d3_c6eb6f4454e0.slice/crio-723f4afe0f7e16bb37170c32dd224cb19668488de0f5636a1d124c9233cfe5ab WatchSource:0}: Error finding container 723f4afe0f7e16bb37170c32dd224cb19668488de0f5636a1d124c9233cfe5ab: Status 404 returned error can't find the container with id 723f4afe0f7e16bb37170c32dd224cb19668488de0f5636a1d124c9233cfe5ab Nov 02 14:38:56.057101 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:56.057074 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" event={"ID":"1d6f069c-113b-42a2-84d3-c6eb6f4454e0","Type":"ContainerStarted","Data":"f69008dab040a6d65aeea0a6a9f7c9ebcce718cb8b7d05565cac2687da5e35a2"} Nov 02 14:38:56.057245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:56.057127 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" event={"ID":"1d6f069c-113b-42a2-84d3-c6eb6f4454e0","Type":"ContainerStarted","Data":"723f4afe0f7e16bb37170c32dd224cb19668488de0f5636a1d124c9233cfe5ab"} Nov 02 14:38:56.110133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:56.110086 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" podStartSLOduration=1.110068766 podStartE2EDuration="1.110068766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:56.105928984 +0000 UTC m=+2501.143033916" watchObservedRunningTime="2026-11-02 14:38:56.110068766 +0000 UTC m=+2501.147173699" Nov 02 14:38:57.064583 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:57.064544 4431 generic.go:356] "Generic (PLEG): container finished" podID="1d6f069c-113b-42a2-84d3-c6eb6f4454e0" containerID="f69008dab040a6d65aeea0a6a9f7c9ebcce718cb8b7d05565cac2687da5e35a2" exitCode=0 Nov 02 14:38:57.064816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:57.064631 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" event={"ID":"1d6f069c-113b-42a2-84d3-c6eb6f4454e0","Type":"ContainerDied","Data":"f69008dab040a6d65aeea0a6a9f7c9ebcce718cb8b7d05565cac2687da5e35a2"} Nov 02 14:38:58.246406 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.246381 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:58.303208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.302275 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-kube-api-access-cksz7\" (UniqueName: \"kubernetes.io/projected/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-kube-api-access-cksz7\") pod \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " Nov 02 14:38:58.303208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.302328 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-xks-ca\") pod \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " Nov 02 14:38:58.303208 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.302431 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-openshift-ca\") pod \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\" (UID: \"1d6f069c-113b-42a2-84d3-c6eb6f4454e0\") " Nov 02 14:38:58.303981 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.303955 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-openshift-ca" pod "1d6f069c-113b-42a2-84d3-c6eb6f4454e0" (UID: "1d6f069c-113b-42a2-84d3-c6eb6f4454e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:38:58.312221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.312176 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-xks-ca" pod "1d6f069c-113b-42a2-84d3-c6eb6f4454e0" (UID: "1d6f069c-113b-42a2-84d3-c6eb6f4454e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:38:58.319797 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.318675 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-kube-api-access-cksz7" pod "1d6f069c-113b-42a2-84d3-c6eb6f4454e0" (UID: "1d6f069c-113b-42a2-84d3-c6eb6f4454e0"). InnerVolumeSpecName "kube-api-access-cksz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:38:58.403800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.403743 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cksz7\" (UniqueName: \"kubernetes.io/projected/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-kube-api-access-cksz7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:58.403800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.403771 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:58.403800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:58.403807 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d6f069c-113b-42a2-84d3-c6eb6f4454e0-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:38:59.080351 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:59.080323 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" Nov 02 14:38:59.080533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:59.080352 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f" event={"ID":"1d6f069c-113b-42a2-84d3-c6eb6f4454e0","Type":"ContainerDied","Data":"723f4afe0f7e16bb37170c32dd224cb19668488de0f5636a1d124c9233cfe5ab"} Nov 02 14:38:59.080533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:38:59.080389 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="723f4afe0f7e16bb37170c32dd224cb19668488de0f5636a1d124c9233cfe5ab" Nov 02 14:39:03.979322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:03.979282 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" containerID="cri-o://aff6593b02f19045c9f525ff184c812324717e0b80b0346aa89d30f242a1a863" gracePeriod=45 Nov 02 14:39:04.114033 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.113985 4431 generic.go:356] "Generic (PLEG): container finished" podID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerID="aff6593b02f19045c9f525ff184c812324717e0b80b0346aa89d30f242a1a863" exitCode=0 Nov 02 14:39:04.114206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.114188 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" event={"ID":"c0236687-f8bf-4b9e-a5fb-cee272e4d86e","Type":"ContainerDied","Data":"aff6593b02f19045c9f525ff184c812324717e0b80b0346aa89d30f242a1a863"} Nov 02 14:39:04.264873 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.264848 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:39:04.301295 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.301256 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn"] Nov 02 14:39:04.302035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.301772 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" containerID="cri-o://243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d" gracePeriod=30 Nov 02 14:39:04.379610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.379569 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-model-cache\") pod \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " Nov 02 14:39:04.379843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.379627 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tls-certs\") pod \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " Nov 02 14:39:04.379843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.379804 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-home\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-home\") pod \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " Nov 02 14:39:04.380026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.379848 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tmp-dir\") pod \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " Nov 02 14:39:04.380026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.379882 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-kube-api-access-m4v2l\" (UniqueName: \"kubernetes.io/projected/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-kube-api-access-m4v2l\") pod \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " Nov 02 14:39:04.380026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.379880 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-model-cache" pod "c0236687-f8bf-4b9e-a5fb-cee272e4d86e" (UID: "c0236687-f8bf-4b9e-a5fb-cee272e4d86e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:04.380026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.379956 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-dshm\") pod \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\" (UID: \"c0236687-f8bf-4b9e-a5fb-cee272e4d86e\") " Nov 02 14:39:04.380306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.380098 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-home" pod "c0236687-f8bf-4b9e-a5fb-cee272e4d86e" (UID: "c0236687-f8bf-4b9e-a5fb-cee272e4d86e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:04.380377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.380308 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tmp-dir" pod "c0236687-f8bf-4b9e-a5fb-cee272e4d86e" (UID: "c0236687-f8bf-4b9e-a5fb-cee272e4d86e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:04.380589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.380511 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.380589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.380530 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.380589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.380544 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.382672 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.382640 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-dshm" pod "c0236687-f8bf-4b9e-a5fb-cee272e4d86e" (UID: "c0236687-f8bf-4b9e-a5fb-cee272e4d86e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:04.382819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.382692 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-kube-api-access-m4v2l" pod "c0236687-f8bf-4b9e-a5fb-cee272e4d86e" (UID: "c0236687-f8bf-4b9e-a5fb-cee272e4d86e"). InnerVolumeSpecName "kube-api-access-m4v2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:04.382922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.382894 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tls-certs" pod "c0236687-f8bf-4b9e-a5fb-cee272e4d86e" (UID: "c0236687-f8bf-4b9e-a5fb-cee272e4d86e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:04.484675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.484575 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.484675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.484606 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:04.484675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:04.484617 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4v2l\" (UniqueName: \"kubernetes.io/projected/c0236687-f8bf-4b9e-a5fb-cee272e4d86e-kube-api-access-m4v2l\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:05.122733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:05.122704 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" Nov 02 14:39:05.122733 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:05.122712 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q" event={"ID":"c0236687-f8bf-4b9e-a5fb-cee272e4d86e","Type":"ContainerDied","Data":"28192a34d2dc26c97f828cb4a3b641644cadc3bc1e00168240ec237c05472caa"} Nov 02 14:39:05.123290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:05.122759 4431 scope.go:122] "RemoveContainer" containerID="aff6593b02f19045c9f525ff184c812324717e0b80b0346aa89d30f242a1a863" Nov 02 14:39:05.180049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:05.180021 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q"] Nov 02 14:39:05.193317 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:05.193258 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-555d1e/simulator-w0-555d1e-kserve-5848449944-xqz6q"] Nov 02 14:39:06.078987 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.078958 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx"] Nov 02 14:39:06.079464 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.079450 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" Nov 02 14:39:06.079526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.079465 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" Nov 02 14:39:06.079526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.079495 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1d6f069c-113b-42a2-84d3-c6eb6f4454e0" containerName="revoke-keys" Nov 02 14:39:06.079526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.079500 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6f069c-113b-42a2-84d3-c6eb6f4454e0" containerName="revoke-keys" Nov 02 14:39:06.079622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.079584 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6f069c-113b-42a2-84d3-c6eb6f4454e0" containerName="revoke-keys" Nov 02 14:39:06.079622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.079599 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" containerName="main" Nov 02 14:39:06.084538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.084520 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.103847 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.103822 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx"] Nov 02 14:39:06.175853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.175816 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:39:06.176247 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.175891 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:39:06.203960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.203926 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.204091 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.204055 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpl84\" (UniqueName: \"kubernetes.io/projected/86046477-7a5b-48c0-8a2c-fe3ef1589481-kube-api-access-wpl84\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.204144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.204113 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.305666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.305639 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpl84\" (UniqueName: \"kubernetes.io/projected/86046477-7a5b-48c0-8a2c-fe3ef1589481-kube-api-access-wpl84\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.305837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.305711 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.305931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.305842 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.306402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.306380 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.308175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.308153 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.330064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.330015 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpl84\" (UniqueName: \"kubernetes.io/projected/86046477-7a5b-48c0-8a2c-fe3ef1589481-kube-api-access-wpl84\") pod \"maas-api-revoke-sub-7348e0ec-8svwx\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.398049 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.398028 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:06.701314 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:06.701276 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx"] Nov 02 14:39:06.702335 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:39:06.702311 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86046477_7a5b_48c0_8a2c_fe3ef1589481.slice/crio-bf480bbe6ef7cf954cf7d1c28cbb7f149ac8a7fbb5202e3d5bdcc3e9db0d792d WatchSource:0}: Error finding container bf480bbe6ef7cf954cf7d1c28cbb7f149ac8a7fbb5202e3d5bdcc3e9db0d792d: Status 404 returned error can't find the container with id bf480bbe6ef7cf954cf7d1c28cbb7f149ac8a7fbb5202e3d5bdcc3e9db0d792d Nov 02 14:39:07.076238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:07.076209 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0236687-f8bf-4b9e-a5fb-cee272e4d86e" path="/var/lib/kubelet/pods/c0236687-f8bf-4b9e-a5fb-cee272e4d86e/volumes" Nov 02 14:39:07.149114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:07.149072 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" event={"ID":"86046477-7a5b-48c0-8a2c-fe3ef1589481","Type":"ContainerStarted","Data":"7519afb0523e1f45015f594c9abe10e86582f4e166844556a7c0529075289ec1"} Nov 02 14:39:07.149290 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:07.149175 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" event={"ID":"86046477-7a5b-48c0-8a2c-fe3ef1589481","Type":"ContainerStarted","Data":"bf480bbe6ef7cf954cf7d1c28cbb7f149ac8a7fbb5202e3d5bdcc3e9db0d792d"} Nov 02 14:39:07.197186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:07.197132 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" podStartSLOduration=1.197117277 podStartE2EDuration="1.197117277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:07.19382441 +0000 UTC m=+2512.230929342" watchObservedRunningTime="2026-11-02 14:39:07.197117277 +0000 UTC m=+2512.234222209" Nov 02 14:39:08.156824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.156760 4431 generic.go:356] "Generic (PLEG): container finished" podID="86046477-7a5b-48c0-8a2c-fe3ef1589481" containerID="7519afb0523e1f45015f594c9abe10e86582f4e166844556a7c0529075289ec1" exitCode=0 Nov 02 14:39:08.156824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.156809 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" event={"ID":"86046477-7a5b-48c0-8a2c-fe3ef1589481","Type":"ContainerDied","Data":"7519afb0523e1f45015f594c9abe10e86582f4e166844556a7c0529075289ec1"} Nov 02 14:39:08.221270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.221234 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:39:08.719701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.719672 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6"] Nov 02 14:39:08.723888 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.723859 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.744446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.744425 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6"] Nov 02 14:39:08.835603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.835570 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r729j\" (UniqueName: \"kubernetes.io/projected/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-kube-api-access-r729j\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.835603 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.835615 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.836023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.836001 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.937194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.937165 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.937385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.937256 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r729j\" (UniqueName: \"kubernetes.io/projected/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-kube-api-access-r729j\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.937385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.937285 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.937991 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.937972 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.939513 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.939492 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:08.962008 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:08.961979 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r729j\" (UniqueName: \"kubernetes.io/projected/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-kube-api-access-r729j\") pod \"maas-api-revoke-sub-f8a9b852-x2cp6\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:09.038224 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.038138 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:09.404117 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:39:09.404086 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ae91ec_a9ab_494c_aea8_b844a8e5a5e4.slice/crio-42915ec5dcad772c0ad27b845ba9dc76b290873e7cbed9a7fb469eae50ae5fc8 WatchSource:0}: Error finding container 42915ec5dcad772c0ad27b845ba9dc76b290873e7cbed9a7fb469eae50ae5fc8: Status 404 returned error can't find the container with id 42915ec5dcad772c0ad27b845ba9dc76b290873e7cbed9a7fb469eae50ae5fc8 Nov 02 14:39:09.404514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.404490 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6"] Nov 02 14:39:09.425277 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.425235 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:09.548987 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.548959 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-openshift-ca\") pod \"86046477-7a5b-48c0-8a2c-fe3ef1589481\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " Nov 02 14:39:09.549175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.549057 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-xks-ca\") pod \"86046477-7a5b-48c0-8a2c-fe3ef1589481\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " Nov 02 14:39:09.549241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.549197 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86046477-7a5b-48c0-8a2c-fe3ef1589481-kube-api-access-wpl84\" (UniqueName: \"kubernetes.io/projected/86046477-7a5b-48c0-8a2c-fe3ef1589481-kube-api-access-wpl84\") pod \"86046477-7a5b-48c0-8a2c-fe3ef1589481\" (UID: \"86046477-7a5b-48c0-8a2c-fe3ef1589481\") " Nov 02 14:39:09.549343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.549315 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-openshift-ca" pod "86046477-7a5b-48c0-8a2c-fe3ef1589481" (UID: "86046477-7a5b-48c0-8a2c-fe3ef1589481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:09.549588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.549571 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.570644 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.570618 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-xks-ca" pod "86046477-7a5b-48c0-8a2c-fe3ef1589481" (UID: "86046477-7a5b-48c0-8a2c-fe3ef1589481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:09.570812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.570654 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86046477-7a5b-48c0-8a2c-fe3ef1589481-kube-api-access-wpl84" pod "86046477-7a5b-48c0-8a2c-fe3ef1589481" (UID: "86046477-7a5b-48c0-8a2c-fe3ef1589481"). InnerVolumeSpecName "kube-api-access-wpl84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:09.581956 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.581907 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:39:09.650081 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650054 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-credential-socket\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650111 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istiod-ca-cert\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650183 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-kube-api-access-9hv4t\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650217 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-socket\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650357 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-data\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650365 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-credential-socket" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:09.650443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650436 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-token\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-token\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650513 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-socket" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:09.650625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650533 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-podinfo\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650575 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-envoy\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650610 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-certs\") pod \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\" (UID: \"83d3dd31-1898-4fdc-9f62-21d736f8ec14\") " Nov 02 14:39:09.650880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.650659 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-data" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:09.651155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.651134 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86046477-7a5b-48c0-8a2c-fe3ef1589481-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.651235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.651161 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.651235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.651175 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.651235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.651190 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpl84\" (UniqueName: \"kubernetes.io/projected/86046477-7a5b-48c0-8a2c-fe3ef1589481-kube-api-access-wpl84\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.651235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.651203 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.651235 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.651214 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-certs" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:09.651526 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.651252 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istiod-ca-cert" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:09.652630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.652603 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-token" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:09.653055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.653032 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-envoy" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:09.653366 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.653343 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-podinfo" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:39:09.653411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.653366 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-kube-api-access-9hv4t" pod "83d3dd31-1898-4fdc-9f62-21d736f8ec14" (UID: "83d3dd31-1898-4fdc-9f62-21d736f8ec14"). InnerVolumeSpecName "kube-api-access-9hv4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:09.751972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.751934 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hv4t\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-kube-api-access-9hv4t\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.751972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.751969 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.751972 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.751979 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.752223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.751988 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.752223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.751996 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d3dd31-1898-4fdc-9f62-21d736f8ec14-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:09.752223 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:09.752003 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d3dd31-1898-4fdc-9f62-21d736f8ec14-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:10.179287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.179256 4431 generic.go:356] "Generic (PLEG): container finished" podID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerID="243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d" exitCode=0 Nov 02 14:39:10.179443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.179293 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" Nov 02 14:39:10.179443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.179343 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" event={"ID":"83d3dd31-1898-4fdc-9f62-21d736f8ec14","Type":"ContainerDied","Data":"243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d"} Nov 02 14:39:10.179443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.179391 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn" event={"ID":"83d3dd31-1898-4fdc-9f62-21d736f8ec14","Type":"ContainerDied","Data":"6da70dc4e16390479b96e2fad158e74e86bb72dfe15471a7876dc6250acff8bc"} Nov 02 14:39:10.179443 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.179414 4431 scope.go:122] "RemoveContainer" containerID="243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d" Nov 02 14:39:10.181292 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.181266 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" event={"ID":"86046477-7a5b-48c0-8a2c-fe3ef1589481","Type":"ContainerDied","Data":"bf480bbe6ef7cf954cf7d1c28cbb7f149ac8a7fbb5202e3d5bdcc3e9db0d792d"} Nov 02 14:39:10.181402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.181352 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf480bbe6ef7cf954cf7d1c28cbb7f149ac8a7fbb5202e3d5bdcc3e9db0d792d" Nov 02 14:39:10.181469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.181420 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx" Nov 02 14:39:10.190472 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.190445 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" event={"ID":"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4","Type":"ContainerStarted","Data":"019d728531959edf8daa215c711f76ddd814e7cb5bd055f631cc0f060dcd6794"} Nov 02 14:39:10.190863 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.190657 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" event={"ID":"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4","Type":"ContainerStarted","Data":"42915ec5dcad772c0ad27b845ba9dc76b290873e7cbed9a7fb469eae50ae5fc8"} Nov 02 14:39:10.192938 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.192917 4431 scope.go:122] "RemoveContainer" containerID="18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8" Nov 02 14:39:10.203648 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.203628 4431 scope.go:122] "RemoveContainer" containerID="243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d" Nov 02 14:39:10.203940 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:39:10.203920 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d\": container with ID starting with 243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d not found: ID does not exist" containerID="243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d" Nov 02 14:39:10.204003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.203954 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d"} err="failed to get container status \"243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d\": rpc error: code = NotFound desc = could not find container \"243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d\": container with ID starting with 243b8cf63a43c6cfd8d4ad5a9428d0052decf42e15efbbb33004c6ba70e8647d not found: ID does not exist" Nov 02 14:39:10.204003 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.203971 4431 scope.go:122] "RemoveContainer" containerID="18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8" Nov 02 14:39:10.204237 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:39:10.204219 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8\": container with ID starting with 18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8 not found: ID does not exist" containerID="18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8" Nov 02 14:39:10.204285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.204241 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8"} err="failed to get container status \"18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8\": rpc error: code = NotFound desc = could not find container \"18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8\": container with ID starting with 18ac3b99ac4810ab6ecdaf5b443a85018f24b0f8e9e52741477ebdfae0e527d8 not found: ID does not exist" Nov 02 14:39:10.236561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.236541 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn"] Nov 02 14:39:10.249370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.249351 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f1e1c2-openshift-default-5b9dc59c9f-7cjbn"] Nov 02 14:39:10.292879 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.292831 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" podStartSLOduration=2.292820906 podStartE2EDuration="2.292820906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:10.289895888 +0000 UTC m=+2515.327000821" watchObservedRunningTime="2026-11-02 14:39:10.292820906 +0000 UTC m=+2515.329925839" Nov 02 14:39:10.876946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.876914 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r"] Nov 02 14:39:10.932306 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:10.932270 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5"] Nov 02 14:39:11.076395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:11.076366 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" path="/var/lib/kubelet/pods/83d3dd31-1898-4fdc-9f62-21d736f8ec14/volumes" Nov 02 14:39:11.200322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:11.200284 4431 generic.go:356] "Generic (PLEG): container finished" podID="67ae91ec-a9ab-494c-aea8-b844a8e5a5e4" containerID="019d728531959edf8daa215c711f76ddd814e7cb5bd055f631cc0f060dcd6794" exitCode=0 Nov 02 14:39:11.200535 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:11.200372 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" event={"ID":"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4","Type":"ContainerDied","Data":"019d728531959edf8daa215c711f76ddd814e7cb5bd055f631cc0f060dcd6794"} Nov 02 14:39:12.353161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.353138 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:12.480552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.480483 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-openshift-ca\") pod \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " Nov 02 14:39:12.480552 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.480542 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-kube-api-access-r729j\" (UniqueName: \"kubernetes.io/projected/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-kube-api-access-r729j\") pod \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " Nov 02 14:39:12.480825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.480565 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-xks-ca\") pod \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\" (UID: \"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4\") " Nov 02 14:39:12.480908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.480830 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-openshift-ca" pod "67ae91ec-a9ab-494c-aea8-b844a8e5a5e4" (UID: "67ae91ec-a9ab-494c-aea8-b844a8e5a5e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:12.482666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.482634 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-xks-ca" pod "67ae91ec-a9ab-494c-aea8-b844a8e5a5e4" (UID: "67ae91ec-a9ab-494c-aea8-b844a8e5a5e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:12.482771 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.482672 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-kube-api-access-r729j" pod "67ae91ec-a9ab-494c-aea8-b844a8e5a5e4" (UID: "67ae91ec-a9ab-494c-aea8-b844a8e5a5e4"). InnerVolumeSpecName "kube-api-access-r729j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:12.581970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.581942 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:12.581970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.581967 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r729j\" (UniqueName: \"kubernetes.io/projected/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-kube-api-access-r729j\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:12.582148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:12.581977 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:13.216055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:13.216025 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" Nov 02 14:39:13.216228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:13.216054 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6" event={"ID":"67ae91ec-a9ab-494c-aea8-b844a8e5a5e4","Type":"ContainerDied","Data":"42915ec5dcad772c0ad27b845ba9dc76b290873e7cbed9a7fb469eae50ae5fc8"} Nov 02 14:39:13.216228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:13.216111 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42915ec5dcad772c0ad27b845ba9dc76b290873e7cbed9a7fb469eae50ae5fc8" Nov 02 14:39:15.877922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:15.877883 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" podUID="b01f4c12-63f6-431c-a9ca-a1205dd8aacf" containerName="payload-processing" containerID="cri-o://ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e" gracePeriod=25 Nov 02 14:39:15.932767 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:15.932733 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" podUID="b5b0080d-e55e-437f-b649-ec4578a9699b" containerName="payload-pre-processing" containerID="cri-o://0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0" gracePeriod=25 Nov 02 14:39:16.582559 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.582520 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v"] Nov 02 14:39:16.583072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583056 4431 cpu_manager.go:414] "containerMap: removing container" podUID="86046477-7a5b-48c0-8a2c-fe3ef1589481" containerName="revoke-keys" Nov 02 14:39:16.583128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583074 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86046477-7a5b-48c0-8a2c-fe3ef1589481" containerName="revoke-keys" Nov 02 14:39:16.583128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583086 4431 cpu_manager.go:414] "containerMap: removing container" podUID="67ae91ec-a9ab-494c-aea8-b844a8e5a5e4" containerName="revoke-keys" Nov 02 14:39:16.583128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583091 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ae91ec-a9ab-494c-aea8-b844a8e5a5e4" containerName="revoke-keys" Nov 02 14:39:16.583128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583128 4431 cpu_manager.go:414] "containerMap: removing container" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" Nov 02 14:39:16.583266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583133 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" Nov 02 14:39:16.583266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583142 4431 cpu_manager.go:414] "containerMap: removing container" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" Nov 02 14:39:16.583266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583147 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" Nov 02 14:39:16.583266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583214 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" Nov 02 14:39:16.583266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583224 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d3dd31-1898-4fdc-9f62-21d736f8ec14" containerName="istio-proxy" Nov 02 14:39:16.583266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583230 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="86046477-7a5b-48c0-8a2c-fe3ef1589481" containerName="revoke-keys" Nov 02 14:39:16.583266 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.583236 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ae91ec-a9ab-494c-aea8-b844a8e5a5e4" containerName="revoke-keys" Nov 02 14:39:16.586674 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.586658 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.607074 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.607047 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v"] Nov 02 14:39:16.616093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.616071 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:39:16.616093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.616085 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:39:16.724963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.724922 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.725126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.724968 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.725126 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.725041 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6sn5\" (UniqueName: \"kubernetes.io/projected/1f95370a-57e4-46ef-865b-f3d00c69ab53-kube-api-access-s6sn5\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.825876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.825840 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.825876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.825881 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.826132 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.826084 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6sn5\" (UniqueName: \"kubernetes.io/projected/1f95370a-57e4-46ef-865b-f3d00c69ab53-kube-api-access-s6sn5\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.826598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.826575 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.828176 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.828157 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.860690 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.860622 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6sn5\" (UniqueName: \"kubernetes.io/projected/1f95370a-57e4-46ef-865b-f3d00c69ab53-kube-api-access-s6sn5\") pod \"maas-api-revoke-sub-c605362e-7tc4v\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:16.899388 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:16.899355 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:17.209508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:17.209479 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v"] Nov 02 14:39:17.210247 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:39:17.210215 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f95370a_57e4_46ef_865b_f3d00c69ab53.slice/crio-17629cb552180c5a84a5bce72ebe1f0b9150428d2119b140baf510d3be330906 WatchSource:0}: Error finding container 17629cb552180c5a84a5bce72ebe1f0b9150428d2119b140baf510d3be330906: Status 404 returned error can't find the container with id 17629cb552180c5a84a5bce72ebe1f0b9150428d2119b140baf510d3be330906 Nov 02 14:39:17.243372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:17.243340 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" event={"ID":"1f95370a-57e4-46ef-865b-f3d00c69ab53","Type":"ContainerStarted","Data":"17629cb552180c5a84a5bce72ebe1f0b9150428d2119b140baf510d3be330906"} Nov 02 14:39:18.251471 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:18.251440 4431 generic.go:356] "Generic (PLEG): container finished" podID="1f95370a-57e4-46ef-865b-f3d00c69ab53" containerID="9c3ae71da29f2db283e695c2af7bcfbb735fc20470e1e2d9a843de8fc131f5e1" exitCode=0 Nov 02 14:39:18.251882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:18.251514 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" event={"ID":"1f95370a-57e4-46ef-865b-f3d00c69ab53","Type":"ContainerDied","Data":"9c3ae71da29f2db283e695c2af7bcfbb735fc20470e1e2d9a843de8fc131f5e1"} Nov 02 14:39:18.576922 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:18.576893 4431 scope.go:122] "RemoveContainer" containerID="c506b2799404da09776c563e803aa6a1807f557d2c2674085b987d554980211c" Nov 02 14:39:18.742980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:18.742945 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" podUID="b01f4c12-63f6-431c-a9ca-a1205dd8aacf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:39:19.402415 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.402393 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:19.556331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.556260 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-xks-ca\") pod \"1f95370a-57e4-46ef-865b-f3d00c69ab53\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " Nov 02 14:39:19.556490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.556357 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f95370a-57e4-46ef-865b-f3d00c69ab53-kube-api-access-s6sn5\" (UniqueName: \"kubernetes.io/projected/1f95370a-57e4-46ef-865b-f3d00c69ab53-kube-api-access-s6sn5\") pod \"1f95370a-57e4-46ef-865b-f3d00c69ab53\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " Nov 02 14:39:19.556490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.556397 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-openshift-ca\") pod \"1f95370a-57e4-46ef-865b-f3d00c69ab53\" (UID: \"1f95370a-57e4-46ef-865b-f3d00c69ab53\") " Nov 02 14:39:19.556827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.556804 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-openshift-ca" pod "1f95370a-57e4-46ef-865b-f3d00c69ab53" (UID: "1f95370a-57e4-46ef-865b-f3d00c69ab53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:19.558501 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.558474 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-xks-ca" pod "1f95370a-57e4-46ef-865b-f3d00c69ab53" (UID: "1f95370a-57e4-46ef-865b-f3d00c69ab53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:19.558725 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.558588 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f95370a-57e4-46ef-865b-f3d00c69ab53-kube-api-access-s6sn5" pod "1f95370a-57e4-46ef-865b-f3d00c69ab53" (UID: "1f95370a-57e4-46ef-865b-f3d00c69ab53"). InnerVolumeSpecName "kube-api-access-s6sn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:19.658254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.658233 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6sn5\" (UniqueName: \"kubernetes.io/projected/1f95370a-57e4-46ef-865b-f3d00c69ab53-kube-api-access-s6sn5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:19.658254 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.658255 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:19.658385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.658265 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f95370a-57e4-46ef-865b-f3d00c69ab53-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:19.752088 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:19.752062 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" podUID="b5b0080d-e55e-437f-b649-ec4578a9699b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:39:20.266185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:20.266157 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" Nov 02 14:39:20.266185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:20.266184 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v" event={"ID":"1f95370a-57e4-46ef-865b-f3d00c69ab53","Type":"ContainerDied","Data":"17629cb552180c5a84a5bce72ebe1f0b9150428d2119b140baf510d3be330906"} Nov 02 14:39:20.266469 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:20.266229 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17629cb552180c5a84a5bce72ebe1f0b9150428d2119b140baf510d3be330906" Nov 02 14:39:21.285875 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:39:21.285830 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:39:28.742969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:28.742928 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" podUID="b01f4c12-63f6-431c-a9ca-a1205dd8aacf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:39:29.751511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:29.751477 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" podUID="b5b0080d-e55e-437f-b649-ec4578a9699b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 14:39:30.137194 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:30.137165 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:39:31.231718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.231684 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms"] Nov 02 14:39:31.232209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.232193 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1f95370a-57e4-46ef-865b-f3d00c69ab53" containerName="revoke-keys" Nov 02 14:39:31.232251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.232211 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f95370a-57e4-46ef-865b-f3d00c69ab53" containerName="revoke-keys" Nov 02 14:39:31.232326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.232316 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f95370a-57e4-46ef-865b-f3d00c69ab53" containerName="revoke-keys" Nov 02 14:39:31.236967 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.236949 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.260155 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.260133 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:39:31.271418 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.271395 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:39:31.275161 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.275137 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms"] Nov 02 14:39:31.294228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.294197 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhjw\" (UniqueName: \"kubernetes.io/projected/5ce1ed78-7686-4253-a226-5aa680fb5b6b-kube-api-access-kvhjw\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.294383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.294363 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.294448 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.294435 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.395759 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.395727 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhjw\" (UniqueName: \"kubernetes.io/projected/5ce1ed78-7686-4253-a226-5aa680fb5b6b-kube-api-access-kvhjw\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.395960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.395863 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.395960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.395927 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.396665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.396640 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.398328 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.398302 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.420998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.420971 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhjw\" (UniqueName: \"kubernetes.io/projected/5ce1ed78-7686-4253-a226-5aa680fb5b6b-kube-api-access-kvhjw\") pod \"maas-api-revoke-sub-c611d23b-mfsms\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.550382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.550313 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:31.885327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:31.885286 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms"] Nov 02 14:39:32.367241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:32.367203 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" event={"ID":"5ce1ed78-7686-4253-a226-5aa680fb5b6b","Type":"ContainerStarted","Data":"b610d90c089ce0c94f4accd0eb9e929bfe36c034a535f8e5c117d95afcd7b4d7"} Nov 02 14:39:32.367647 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:32.367261 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" event={"ID":"5ce1ed78-7686-4253-a226-5aa680fb5b6b","Type":"ContainerStarted","Data":"ec33cb4eebdc97ed27c2e17aaf84ebe1aac34a7f547074686ea20cf836d60549"} Nov 02 14:39:32.431855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:32.431754 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" podStartSLOduration=1.4317407690000001 podStartE2EDuration="1.431740769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:32.429693967 +0000 UTC m=+2537.466798900" watchObservedRunningTime="2026-11-02 14:39:32.431740769 +0000 UTC m=+2537.468845730" Nov 02 14:39:33.374486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:33.374454 4431 generic.go:356] "Generic (PLEG): container finished" podID="5ce1ed78-7686-4253-a226-5aa680fb5b6b" containerID="b610d90c089ce0c94f4accd0eb9e929bfe36c034a535f8e5c117d95afcd7b4d7" exitCode=0 Nov 02 14:39:33.374897 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:33.374537 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" event={"ID":"5ce1ed78-7686-4253-a226-5aa680fb5b6b","Type":"ContainerDied","Data":"b610d90c089ce0c94f4accd0eb9e929bfe36c034a535f8e5c117d95afcd7b4d7"} Nov 02 14:39:34.598610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.596201 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:34.738511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.738481 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce1ed78-7686-4253-a226-5aa680fb5b6b-kube-api-access-kvhjw\" (UniqueName: \"kubernetes.io/projected/5ce1ed78-7686-4253-a226-5aa680fb5b6b-kube-api-access-kvhjw\") pod \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " Nov 02 14:39:34.738667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.738522 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-openshift-ca\") pod \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " Nov 02 14:39:34.738667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.738621 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-xks-ca\") pod \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\" (UID: \"5ce1ed78-7686-4253-a226-5aa680fb5b6b\") " Nov 02 14:39:34.738977 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.738951 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-openshift-ca" pod "5ce1ed78-7686-4253-a226-5aa680fb5b6b" (UID: "5ce1ed78-7686-4253-a226-5aa680fb5b6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:34.739083 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.739068 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:34.740620 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.740593 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-xks-ca" pod "5ce1ed78-7686-4253-a226-5aa680fb5b6b" (UID: "5ce1ed78-7686-4253-a226-5aa680fb5b6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:34.740824 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.740803 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce1ed78-7686-4253-a226-5aa680fb5b6b-kube-api-access-kvhjw" pod "5ce1ed78-7686-4253-a226-5aa680fb5b6b" (UID: "5ce1ed78-7686-4253-a226-5aa680fb5b6b"). InnerVolumeSpecName "kube-api-access-kvhjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:34.840158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.840125 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvhjw\" (UniqueName: \"kubernetes.io/projected/5ce1ed78-7686-4253-a226-5aa680fb5b6b-kube-api-access-kvhjw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:34.840158 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:34.840152 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce1ed78-7686-4253-a226-5aa680fb5b6b-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:35.091547 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:35.091518 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28"] Nov 02 14:39:35.095702 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:35.095682 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cab08d4-mqf28"] Nov 02 14:39:35.391577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:35.391544 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" event={"ID":"5ce1ed78-7686-4253-a226-5aa680fb5b6b","Type":"ContainerDied","Data":"ec33cb4eebdc97ed27c2e17aaf84ebe1aac34a7f547074686ea20cf836d60549"} Nov 02 14:39:35.391801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:35.391601 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec33cb4eebdc97ed27c2e17aaf84ebe1aac34a7f547074686ea20cf836d60549" Nov 02 14:39:35.391801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:35.391552 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms" Nov 02 14:39:35.803749 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:35.803678 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8"] Nov 02 14:39:35.820895 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:35.820865 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj"] Nov 02 14:39:36.197077 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.197054 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:39:36.222798 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.222762 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:39:36.356347 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.356267 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-plugins-config-volume\") pod \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\" (UID: \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\") " Nov 02 14:39:36.356517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.356380 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b0080d-e55e-437f-b649-ec4578a9699b-kube-api-access-bqf5h\" (UniqueName: \"kubernetes.io/projected/b5b0080d-e55e-437f-b649-ec4578a9699b-kube-api-access-bqf5h\") pod \"b5b0080d-e55e-437f-b649-ec4578a9699b\" (UID: \"b5b0080d-e55e-437f-b649-ec4578a9699b\") " Nov 02 14:39:36.356589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.356544 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b0080d-e55e-437f-b649-ec4578a9699b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b0080d-e55e-437f-b649-ec4578a9699b-plugins-config-volume\") pod \"b5b0080d-e55e-437f-b649-ec4578a9699b\" (UID: \"b5b0080d-e55e-437f-b649-ec4578a9699b\") " Nov 02 14:39:36.356589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.356580 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-kube-api-access-t9gnv\" (UniqueName: \"kubernetes.io/projected/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-kube-api-access-t9gnv\") pod \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\" (UID: \"b01f4c12-63f6-431c-a9ca-a1205dd8aacf\") " Nov 02 14:39:36.356703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.356607 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-plugins-config-volume" pod "b01f4c12-63f6-431c-a9ca-a1205dd8aacf" (UID: "b01f4c12-63f6-431c-a9ca-a1205dd8aacf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:36.356966 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.356936 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b0080d-e55e-437f-b649-ec4578a9699b-plugins-config-volume" pod "b5b0080d-e55e-437f-b649-ec4578a9699b" (UID: "b5b0080d-e55e-437f-b649-ec4578a9699b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:36.357066 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.357031 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:36.358514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.358485 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b0080d-e55e-437f-b649-ec4578a9699b-kube-api-access-bqf5h" pod "b5b0080d-e55e-437f-b649-ec4578a9699b" (UID: "b5b0080d-e55e-437f-b649-ec4578a9699b"). InnerVolumeSpecName "kube-api-access-bqf5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:36.358622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.358562 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-kube-api-access-t9gnv" pod "b01f4c12-63f6-431c-a9ca-a1205dd8aacf" (UID: "b01f4c12-63f6-431c-a9ca-a1205dd8aacf"). InnerVolumeSpecName "kube-api-access-t9gnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:36.398667 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.398643 4431 generic.go:356] "Generic (PLEG): container finished" podID="b01f4c12-63f6-431c-a9ca-a1205dd8aacf" containerID="ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e" exitCode=0 Nov 02 14:39:36.398829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.398698 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" Nov 02 14:39:36.398829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.398733 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" event={"ID":"b01f4c12-63f6-431c-a9ca-a1205dd8aacf","Type":"ContainerDied","Data":"ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e"} Nov 02 14:39:36.398963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.398827 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r" event={"ID":"b01f4c12-63f6-431c-a9ca-a1205dd8aacf","Type":"ContainerDied","Data":"9bd7042fa7a7fc50f291ad336dc0aafe62ef3a63d6cea82b4c0492cc96f8d2e1"} Nov 02 14:39:36.398963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.398866 4431 scope.go:122] "RemoveContainer" containerID="ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e" Nov 02 14:39:36.400185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.400163 4431 generic.go:356] "Generic (PLEG): container finished" podID="b5b0080d-e55e-437f-b649-ec4578a9699b" containerID="0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0" exitCode=0 Nov 02 14:39:36.400283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.400214 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" Nov 02 14:39:36.400283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.400255 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" event={"ID":"b5b0080d-e55e-437f-b649-ec4578a9699b","Type":"ContainerDied","Data":"0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0"} Nov 02 14:39:36.400392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.400303 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5" event={"ID":"b5b0080d-e55e-437f-b649-ec4578a9699b","Type":"ContainerDied","Data":"d14f6234bfde2da8b9ded1db1222e3f3674074ed0ef4a96b30df49c80039de25"} Nov 02 14:39:36.409982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.409961 4431 scope.go:122] "RemoveContainer" containerID="ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e" Nov 02 14:39:36.410259 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:39:36.410232 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e\": container with ID starting with ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e not found: ID does not exist" containerID="ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e" Nov 02 14:39:36.410322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.410272 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e"} err="failed to get container status \"ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e\": rpc error: code = NotFound desc = could not find container \"ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e\": container with ID starting with ab8ea1201d54a27e77b5d1fcb4901b280c74dde6cbf66e2f4df3f200f7981e2e not found: ID does not exist" Nov 02 14:39:36.410322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.410297 4431 scope.go:122] "RemoveContainer" containerID="0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0" Nov 02 14:39:36.419411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.419396 4431 scope.go:122] "RemoveContainer" containerID="0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0" Nov 02 14:39:36.419660 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:39:36.419643 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0\": container with ID starting with 0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0 not found: ID does not exist" containerID="0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0" Nov 02 14:39:36.419714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.419666 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0"} err="failed to get container status \"0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0\": rpc error: code = NotFound desc = could not find container \"0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0\": container with ID starting with 0d09eaa0be888d30a8e32c30983ec7c3b85acc7e4c008f4cb189d686b4a36ec0 not found: ID does not exist" Nov 02 14:39:36.457034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.457012 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r"] Nov 02 14:39:36.457948 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.457925 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqf5h\" (UniqueName: \"kubernetes.io/projected/b5b0080d-e55e-437f-b649-ec4578a9699b-kube-api-access-bqf5h\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:36.458026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.457951 4431 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b0080d-e55e-437f-b649-ec4578a9699b-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:36.458026 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.457967 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9gnv\" (UniqueName: \"kubernetes.io/projected/b01f4c12-63f6-431c-a9ca-a1205dd8aacf-kube-api-access-t9gnv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:36.464864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.464842 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-623b4b-5dcc7b576f-w768r"] Nov 02 14:39:36.505488 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.505464 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5"] Nov 02 14:39:36.510982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:36.510961 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-623b4b-774f7fb84c-kzrt5"] Nov 02 14:39:37.075330 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:37.075299 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd" path="/var/lib/kubelet/pods/98dfb9c1-b5bb-486d-9c8a-0a48b54ac8fd/volumes" Nov 02 14:39:37.075694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:37.075685 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01f4c12-63f6-431c-a9ca-a1205dd8aacf" path="/var/lib/kubelet/pods/b01f4c12-63f6-431c-a9ca-a1205dd8aacf/volumes" Nov 02 14:39:37.076067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:37.076053 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b0080d-e55e-437f-b649-ec4578a9699b" path="/var/lib/kubelet/pods/b5b0080d-e55e-437f-b649-ec4578a9699b/volumes" Nov 02 14:39:46.044893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:46.044860 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b"] Nov 02 14:39:46.058237 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:46.058205 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2fa91b2-rfp7b"] Nov 02 14:39:47.081936 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:47.081899 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df89671-e602-4498-a23f-eeb99eff061c" path="/var/lib/kubelet/pods/1df89671-e602-4498-a23f-eeb99eff061c/volumes" Nov 02 14:39:48.736736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.736703 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm"] Nov 02 14:39:48.737232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737216 4431 cpu_manager.go:414] "containerMap: removing container" podUID="5ce1ed78-7686-4253-a226-5aa680fb5b6b" containerName="revoke-keys" Nov 02 14:39:48.737278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737233 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce1ed78-7686-4253-a226-5aa680fb5b6b" containerName="revoke-keys" Nov 02 14:39:48.737278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737252 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b5b0080d-e55e-437f-b649-ec4578a9699b" containerName="payload-pre-processing" Nov 02 14:39:48.737278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737259 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b0080d-e55e-437f-b649-ec4578a9699b" containerName="payload-pre-processing" Nov 02 14:39:48.737420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737285 4431 cpu_manager.go:414] "containerMap: removing container" podUID="b01f4c12-63f6-431c-a9ca-a1205dd8aacf" containerName="payload-processing" Nov 02 14:39:48.737420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737290 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01f4c12-63f6-431c-a9ca-a1205dd8aacf" containerName="payload-processing" Nov 02 14:39:48.737420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737363 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce1ed78-7686-4253-a226-5aa680fb5b6b" containerName="revoke-keys" Nov 02 14:39:48.737420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737373 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b0080d-e55e-437f-b649-ec4578a9699b" containerName="payload-pre-processing" Nov 02 14:39:48.737420 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.737384 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01f4c12-63f6-431c-a9ca-a1205dd8aacf" containerName="payload-processing" Nov 02 14:39:48.740800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.740757 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.762162 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.762139 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm"] Nov 02 14:39:48.775952 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.775915 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:39:48.776124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.775983 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:39:48.800578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.799645 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.800578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.799709 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvsx\" (UniqueName: \"kubernetes.io/projected/0b585214-c2fc-4bb8-b1d7-54e12b0667be-kube-api-access-dwvsx\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.800578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.799860 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.900860 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.900819 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.901094 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.900875 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvsx\" (UniqueName: \"kubernetes.io/projected/0b585214-c2fc-4bb8-b1d7-54e12b0667be-kube-api-access-dwvsx\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.901094 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.900967 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.901593 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.901569 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.903276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.903257 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:48.925713 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:48.925694 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvsx\" (UniqueName: \"kubernetes.io/projected/0b585214-c2fc-4bb8-b1d7-54e12b0667be-kube-api-access-dwvsx\") pod \"maas-api-revoke-sub-448b04ea-lv8vm\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:49.055599 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:49.055525 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:49.383259 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:49.374580 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm"] Nov 02 14:39:49.499459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:49.499432 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" event={"ID":"0b585214-c2fc-4bb8-b1d7-54e12b0667be","Type":"ContainerStarted","Data":"dc2ab7f592a5c69538e516cf3124d5479f53a74198f2bd0420e52d5fbb4ee82f"} Nov 02 14:39:50.507192 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:50.507165 4431 generic.go:356] "Generic (PLEG): container finished" podID="0b585214-c2fc-4bb8-b1d7-54e12b0667be" containerID="86e08d66a51c371307d11e0ad3c57d57c0f59c45f862e01f263c62bc374e172d" exitCode=0 Nov 02 14:39:50.507520 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:50.507213 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" event={"ID":"0b585214-c2fc-4bb8-b1d7-54e12b0667be","Type":"ContainerDied","Data":"86e08d66a51c371307d11e0ad3c57d57c0f59c45f862e01f263c62bc374e172d"} Nov 02 14:39:50.824538 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:50.824430 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" containerID="cri-o://3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0" gracePeriod=45 Nov 02 14:39:50.841165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:50.841123 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" containerID="cri-o://c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3" gracePeriod=45 Nov 02 14:39:51.112740 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.112716 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:39:51.162013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.161985 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:39:51.225187 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225155 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-model-cache\") pod \"e43d8539-443f-42d1-b09f-7ec8714c1137\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " Nov 02 14:39:51.225362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225205 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-model-cache\") pod \"d28d6443-555d-4b37-b286-ccd55a957b01\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " Nov 02 14:39:51.225362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225240 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-dshm\") pod \"d28d6443-555d-4b37-b286-ccd55a957b01\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " Nov 02 14:39:51.225362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225284 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-tmp-dir\") pod \"e43d8539-443f-42d1-b09f-7ec8714c1137\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " Nov 02 14:39:51.225362 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225333 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-tmp-dir\") pod \"d28d6443-555d-4b37-b286-ccd55a957b01\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " Nov 02 14:39:51.225506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225362 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-home\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-home\") pod \"d28d6443-555d-4b37-b286-ccd55a957b01\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " Nov 02 14:39:51.225506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225445 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-model-cache" pod "e43d8539-443f-42d1-b09f-7ec8714c1137" (UID: "e43d8539-443f-42d1-b09f-7ec8714c1137"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.225506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225474 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28d6443-555d-4b37-b286-ccd55a957b01-kube-api-access-9hw6r\" (UniqueName: \"kubernetes.io/projected/d28d6443-555d-4b37-b286-ccd55a957b01-kube-api-access-9hw6r\") pod \"d28d6443-555d-4b37-b286-ccd55a957b01\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " Nov 02 14:39:51.225602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225511 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43d8539-443f-42d1-b09f-7ec8714c1137-kube-api-access-l8lfp\" (UniqueName: \"kubernetes.io/projected/e43d8539-443f-42d1-b09f-7ec8714c1137-kube-api-access-l8lfp\") pod \"e43d8539-443f-42d1-b09f-7ec8714c1137\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " Nov 02 14:39:51.225602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225553 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-dshm\") pod \"e43d8539-443f-42d1-b09f-7ec8714c1137\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " Nov 02 14:39:51.225602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225585 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43d8539-443f-42d1-b09f-7ec8714c1137-tls-certs\" (UniqueName: \"kubernetes.io/secret/e43d8539-443f-42d1-b09f-7ec8714c1137-tls-certs\") pod \"e43d8539-443f-42d1-b09f-7ec8714c1137\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " Nov 02 14:39:51.225697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225614 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-home\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-home\") pod \"e43d8539-443f-42d1-b09f-7ec8714c1137\" (UID: \"e43d8539-443f-42d1-b09f-7ec8714c1137\") " Nov 02 14:39:51.225697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225617 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-tmp-dir" pod "e43d8539-443f-42d1-b09f-7ec8714c1137" (UID: "e43d8539-443f-42d1-b09f-7ec8714c1137"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.225697 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225653 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28d6443-555d-4b37-b286-ccd55a957b01-tls-certs\" (UniqueName: \"kubernetes.io/secret/d28d6443-555d-4b37-b286-ccd55a957b01-tls-certs\") pod \"d28d6443-555d-4b37-b286-ccd55a957b01\" (UID: \"d28d6443-555d-4b37-b286-ccd55a957b01\") " Nov 02 14:39:51.225881 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.225801 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-model-cache" pod "d28d6443-555d-4b37-b286-ccd55a957b01" (UID: "d28d6443-555d-4b37-b286-ccd55a957b01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.226197 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.226169 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.226322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.226198 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.226322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.226214 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.226685 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.226447 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-home" pod "e43d8539-443f-42d1-b09f-7ec8714c1137" (UID: "e43d8539-443f-42d1-b09f-7ec8714c1137"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.227689 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.227660 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-home" pod "d28d6443-555d-4b37-b286-ccd55a957b01" (UID: "d28d6443-555d-4b37-b286-ccd55a957b01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.227925 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.227899 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-tmp-dir" pod "d28d6443-555d-4b37-b286-ccd55a957b01" (UID: "d28d6443-555d-4b37-b286-ccd55a957b01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.228908 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.228867 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43d8539-443f-42d1-b09f-7ec8714c1137-kube-api-access-l8lfp" pod "e43d8539-443f-42d1-b09f-7ec8714c1137" (UID: "e43d8539-443f-42d1-b09f-7ec8714c1137"). InnerVolumeSpecName "kube-api-access-l8lfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:51.229175 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.229110 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43d8539-443f-42d1-b09f-7ec8714c1137-tls-certs" pod "e43d8539-443f-42d1-b09f-7ec8714c1137" (UID: "e43d8539-443f-42d1-b09f-7ec8714c1137"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:51.230974 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.230941 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28d6443-555d-4b37-b286-ccd55a957b01-tls-certs" pod "d28d6443-555d-4b37-b286-ccd55a957b01" (UID: "d28d6443-555d-4b37-b286-ccd55a957b01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:51.231104 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.231079 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28d6443-555d-4b37-b286-ccd55a957b01-kube-api-access-9hw6r" pod "d28d6443-555d-4b37-b286-ccd55a957b01" (UID: "d28d6443-555d-4b37-b286-ccd55a957b01"). InnerVolumeSpecName "kube-api-access-9hw6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:51.231531 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.231490 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-dshm" pod "e43d8539-443f-42d1-b09f-7ec8714c1137" (UID: "e43d8539-443f-42d1-b09f-7ec8714c1137"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.231722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.231704 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-dshm" pod "d28d6443-555d-4b37-b286-ccd55a957b01" (UID: "d28d6443-555d-4b37-b286-ccd55a957b01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:39:51.286498 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:39:51.286462 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:39:51.327392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327363 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hw6r\" (UniqueName: \"kubernetes.io/projected/d28d6443-555d-4b37-b286-ccd55a957b01-kube-api-access-9hw6r\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.327392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327389 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8lfp\" (UniqueName: \"kubernetes.io/projected/e43d8539-443f-42d1-b09f-7ec8714c1137-kube-api-access-l8lfp\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.327570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327398 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.327570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327412 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43d8539-443f-42d1-b09f-7ec8714c1137-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.327570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327422 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43d8539-443f-42d1-b09f-7ec8714c1137-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.327570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327430 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d28d6443-555d-4b37-b286-ccd55a957b01-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.327570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327437 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.327570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327444 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.327570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.327451 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d28d6443-555d-4b37-b286-ccd55a957b01-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.515032 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.515000 4431 generic.go:356] "Generic (PLEG): container finished" podID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerID="c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3" exitCode=0 Nov 02 14:39:51.515459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.515056 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" Nov 02 14:39:51.515459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.515082 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" event={"ID":"e43d8539-443f-42d1-b09f-7ec8714c1137","Type":"ContainerDied","Data":"c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3"} Nov 02 14:39:51.515459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.515123 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj" event={"ID":"e43d8539-443f-42d1-b09f-7ec8714c1137","Type":"ContainerDied","Data":"69fbdcf72ebd9355de5122a5250e00d6604ae644f319f07485b4bf429039ecd5"} Nov 02 14:39:51.515459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.515139 4431 scope.go:122] "RemoveContainer" containerID="c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3" Nov 02 14:39:51.516700 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.516677 4431 generic.go:356] "Generic (PLEG): container finished" podID="d28d6443-555d-4b37-b286-ccd55a957b01" containerID="3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0" exitCode=0 Nov 02 14:39:51.516855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.516732 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" Nov 02 14:39:51.516855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.516805 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" event={"ID":"d28d6443-555d-4b37-b286-ccd55a957b01","Type":"ContainerDied","Data":"3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0"} Nov 02 14:39:51.516855 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.516840 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8" event={"ID":"d28d6443-555d-4b37-b286-ccd55a957b01","Type":"ContainerDied","Data":"a9cb77eb3337dfd8686d65e95551858a985b88eefc3c33c53e012cac36c1bd51"} Nov 02 14:39:51.526338 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.526318 4431 scope.go:122] "RemoveContainer" containerID="c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3" Nov 02 14:39:51.526882 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:39:51.526605 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3\": container with ID starting with c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3 not found: ID does not exist" containerID="c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3" Nov 02 14:39:51.526882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.526639 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3"} err="failed to get container status \"c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3\": rpc error: code = NotFound desc = could not find container \"c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3\": container with ID starting with c2907297ea7a02be329c0088310dfe11bda1715043cd2269427f23095e7f3db3 not found: ID does not exist" Nov 02 14:39:51.526882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.526662 4431 scope.go:122] "RemoveContainer" containerID="3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0" Nov 02 14:39:51.538426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.538408 4431 scope.go:122] "RemoveContainer" containerID="3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0" Nov 02 14:39:51.538802 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:39:51.538754 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0\": container with ID starting with 3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0 not found: ID does not exist" containerID="3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0" Nov 02 14:39:51.538890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.538804 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0"} err="failed to get container status \"3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0\": rpc error: code = NotFound desc = could not find container \"3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0\": container with ID starting with 3a228e7bd6b07de5471dc6b62c5a0c94df630989b17fc245e73e6b14c7dea9f0 not found: ID does not exist" Nov 02 14:39:51.572085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.572060 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8"] Nov 02 14:39:51.576075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.576054 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/distinct-2-w4-c0c28f-kserve-59d5898c9d-pfzk8"] Nov 02 14:39:51.614714 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.614690 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj"] Nov 02 14:39:51.622819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.622793 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c0c28f/premium-w4-c0c28f-kserve-7b54844d86-wg5fj"] Nov 02 14:39:51.647261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.647237 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:51.731710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.731685 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-xks-ca\") pod \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " Nov 02 14:39:51.732020 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.731994 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b585214-c2fc-4bb8-b1d7-54e12b0667be-kube-api-access-dwvsx\" (UniqueName: \"kubernetes.io/projected/0b585214-c2fc-4bb8-b1d7-54e12b0667be-kube-api-access-dwvsx\") pod \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " Nov 02 14:39:51.732107 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.732066 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-openshift-ca\") pod \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\" (UID: \"0b585214-c2fc-4bb8-b1d7-54e12b0667be\") " Nov 02 14:39:51.732436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.732412 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-openshift-ca" pod "0b585214-c2fc-4bb8-b1d7-54e12b0667be" (UID: "0b585214-c2fc-4bb8-b1d7-54e12b0667be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:39:51.734340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.734311 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-xks-ca" pod "0b585214-c2fc-4bb8-b1d7-54e12b0667be" (UID: "0b585214-c2fc-4bb8-b1d7-54e12b0667be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:39:51.734436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.734333 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.734436 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.734352 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b585214-c2fc-4bb8-b1d7-54e12b0667be-kube-api-access-dwvsx" pod "0b585214-c2fc-4bb8-b1d7-54e12b0667be" (UID: "0b585214-c2fc-4bb8-b1d7-54e12b0667be"). InnerVolumeSpecName "kube-api-access-dwvsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:39:51.834973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.834911 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b585214-c2fc-4bb8-b1d7-54e12b0667be-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:51.834973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:51.834933 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwvsx\" (UniqueName: \"kubernetes.io/projected/0b585214-c2fc-4bb8-b1d7-54e12b0667be-kube-api-access-dwvsx\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:39:52.526850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:52.526818 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" event={"ID":"0b585214-c2fc-4bb8-b1d7-54e12b0667be","Type":"ContainerDied","Data":"dc2ab7f592a5c69538e516cf3124d5479f53a74198f2bd0420e52d5fbb4ee82f"} Nov 02 14:39:52.527255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:52.526870 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc2ab7f592a5c69538e516cf3124d5479f53a74198f2bd0420e52d5fbb4ee82f" Nov 02 14:39:52.527255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:52.526830 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm" Nov 02 14:39:53.075637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:53.075606 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" path="/var/lib/kubelet/pods/d28d6443-555d-4b37-b286-ccd55a957b01/volumes" Nov 02 14:39:53.076058 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:53.076044 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" path="/var/lib/kubelet/pods/e43d8539-443f-42d1-b09f-7ec8714c1137/volumes" Nov 02 14:39:59.657482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:39:59.657406 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:40:05.038474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:05.038433 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v"] Nov 02 14:40:05.051309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:05.051278 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ce61dad-bj85v"] Nov 02 14:40:05.076168 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:05.076135 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756424eb-a5bd-444f-8408-dad53396ee4f" path="/var/lib/kubelet/pods/756424eb-a5bd-444f-8408-dad53396ee4f/volumes" Nov 02 14:40:17.035743 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:17.035705 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n"] Nov 02 14:40:17.048569 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:17.048538 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a045-gpv8n"] Nov 02 14:40:17.075880 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:17.075844 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fd92b6-8de4-4322-ae07-d21439ce7e6a" path="/var/lib/kubelet/pods/e5fd92b6-8de4-4322-ae07-d21439ce7e6a/volumes" Nov 02 14:40:18.616287 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:18.616257 4431 scope.go:122] "RemoveContainer" containerID="335cbd6c98a9056078217e7e6303d5b3d754d5e3b8482c79cb9cb8f78a10aad7" Nov 02 14:40:19.639357 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:19.639317 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:40:21.288574 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:40:21.288531 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:40:22.511055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:22.511020 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj"] Nov 02 14:40:22.511571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:22.511250 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" containerID="cri-o://f8cba83e45358de3705b49f107e2b7fae4ac0ab8a95adea159bcfa806bfdbef2" gracePeriod=30 Nov 02 14:40:27.772708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.772680 4431 generic.go:356] "Generic (PLEG): container finished" podID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerID="f8cba83e45358de3705b49f107e2b7fae4ac0ab8a95adea159bcfa806bfdbef2" exitCode=0 Nov 02 14:40:27.773138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.772763 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" event={"ID":"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4","Type":"ContainerDied","Data":"f8cba83e45358de3705b49f107e2b7fae4ac0ab8a95adea159bcfa806bfdbef2"} Nov 02 14:40:27.773138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.772859 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" event={"ID":"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4","Type":"ContainerDied","Data":"7c5deea707e2e216c454f8fcf5d28472919d6f22ae584fca8e92b01e60b4f3ef"} Nov 02 14:40:27.773138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.772883 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c5deea707e2e216c454f8fcf5d28472919d6f22ae584fca8e92b01e60b4f3ef" Nov 02 14:40:27.783809 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.783734 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:40:27.869630 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869607 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-certs\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.869796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869642 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-envoy\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.869796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869667 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-podinfo\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.869796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869732 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-kube-api-access-x4gnr\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-kube-api-access-x4gnr\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.869796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869761 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-credential-socket\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.870053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869830 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-socket\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.870053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869878 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istiod-ca-cert\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.870053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869919 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-data\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.870053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869952 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-certs" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:27.870053 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.869975 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-token\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-token\") pod \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\" (UID: \"f5edc2bd-c7dd-40b0-9e46-db8efd1428b4\") " Nov 02 14:40:27.870327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.870152 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-socket" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:27.870392 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.870364 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-credential-socket" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:27.870452 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.870394 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istiod-ca-cert" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:40:27.870594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.870567 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-data" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:27.870594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.870586 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:27.870730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.870607 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:27.870730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.870621 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:27.870730 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.870633 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:27.872852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.872814 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-podinfo" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:40:27.872969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.872905 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-kube-api-access-x4gnr" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "kube-api-access-x4gnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:27.873125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.873100 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-envoy" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:27.875485 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.875460 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-token" pod "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" (UID: "f5edc2bd-c7dd-40b0-9e46-db8efd1428b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:27.972331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.972305 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:27.972331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.972328 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:27.972487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.972337 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:27.972487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.972345 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:27.972487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:27.972353 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4gnr\" (UniqueName: \"kubernetes.io/projected/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4-kube-api-access-x4gnr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:28.779226 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:28.779196 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj" Nov 02 14:40:28.866913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:28.866886 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj"] Nov 02 14:40:28.879804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:28.879769 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c0c28f-openshift-default-78946b8b89-97hfj"] Nov 02 14:40:29.076432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:29.076363 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" path="/var/lib/kubelet/pods/f5edc2bd-c7dd-40b0-9e46-db8efd1428b4/volumes" Nov 02 14:40:50.899746 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:50.899710 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb"] Nov 02 14:40:50.900762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:50.900706 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" containerID="cri-o://4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14" gracePeriod=30 Nov 02 14:40:51.135509 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:51.135480 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:40:51.283215 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:40:51.283137 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:40:56.165806 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.165756 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:40:56.272508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272435 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-socket\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.272659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272522 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istiod-ca-cert\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.272659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272563 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-data\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.272659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272585 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-token\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-token\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.272659 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272643 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-envoy\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.273215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272726 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-socket" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:56.273215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272791 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-data" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:56.273215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272768 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-kube-api-access-t8ztz\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-kube-api-access-t8ztz\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.273215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272933 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-certs\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.273215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272962 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-credential-socket\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.273215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.272983 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istiod-ca-cert" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:40:56.273215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.273019 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-podinfo\") pod \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\" (UID: \"3eb08832-f1bd-44e1-bf69-b9d3a298980b\") " Nov 02 14:40:56.273215 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.273190 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-certs" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:56.273597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.273289 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-credential-socket" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:56.273597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.273508 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.273597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.273529 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.273597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.273544 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.273597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.273559 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.273597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.273573 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.274955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.274921 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-token" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:56.275092 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.275006 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-envoy" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:40:56.275460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.275440 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-kube-api-access-t8ztz" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "kube-api-access-t8ztz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:40:56.275460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.275443 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-podinfo" pod "3eb08832-f1bd-44e1-bf69-b9d3a298980b" (UID: "3eb08832-f1bd-44e1-bf69-b9d3a298980b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:40:56.375006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.374979 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.375006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.375005 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8ztz\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-kube-api-access-t8ztz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.375184 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.375015 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.375184 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.375024 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb08832-f1bd-44e1-bf69-b9d3a298980b-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:40:56.968719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.968686 4431 generic.go:356] "Generic (PLEG): container finished" podID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerID="4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14" exitCode=0 Nov 02 14:40:56.968903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.968741 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" Nov 02 14:40:56.968903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.968801 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" event={"ID":"3eb08832-f1bd-44e1-bf69-b9d3a298980b","Type":"ContainerDied","Data":"4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14"} Nov 02 14:40:56.968903 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.968889 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb" event={"ID":"3eb08832-f1bd-44e1-bf69-b9d3a298980b","Type":"ContainerDied","Data":"7669f44a8cc3a3afc55f78c62953a70bd221972ac91693fc261edd6f4ad59ded"} Nov 02 14:40:56.969067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.968917 4431 scope.go:122] "RemoveContainer" containerID="4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14" Nov 02 14:40:56.979349 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.979334 4431 scope.go:122] "RemoveContainer" containerID="a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c" Nov 02 14:40:56.988140 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.988124 4431 scope.go:122] "RemoveContainer" containerID="4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14" Nov 02 14:40:56.988393 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:40:56.988371 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14\": container with ID starting with 4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14 not found: ID does not exist" containerID="4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14" Nov 02 14:40:56.988440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.988402 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14"} err="failed to get container status \"4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14\": rpc error: code = NotFound desc = could not find container \"4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14\": container with ID starting with 4e3928a29cbcc832fb990bbf4d4a0967975f1a58a027deb9fb7709d7eb78ff14 not found: ID does not exist" Nov 02 14:40:56.988440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.988420 4431 scope.go:122] "RemoveContainer" containerID="a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c" Nov 02 14:40:56.988639 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:40:56.988621 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c\": container with ID starting with a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c not found: ID does not exist" containerID="a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c" Nov 02 14:40:56.988702 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:56.988650 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c"} err="failed to get container status \"a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c\": rpc error: code = NotFound desc = could not find container \"a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c\": container with ID starting with a17c6f07fb2ca48a9ce7f4dd79e6d787a1d1f435fc0b467f64f3d87fd98d994c not found: ID does not exist" Nov 02 14:40:57.044748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:57.044719 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb"] Nov 02 14:40:57.054144 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:57.054121 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e07583-openshift-default-6554fb6f65-rgcfb"] Nov 02 14:40:57.075072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:40:57.075048 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" path="/var/lib/kubelet/pods/3eb08832-f1bd-44e1-bf69-b9d3a298980b/volumes" Nov 02 14:41:05.036971 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:05.036941 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn"] Nov 02 14:41:05.050136 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:05.050106 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d67b4b-27tdn"] Nov 02 14:41:05.074866 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:05.074840 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195" path="/var/lib/kubelet/pods/20b27e3b-f0ad-44ce-8bf6-ce8ed08ab195/volumes" Nov 02 14:41:06.628720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:06.628688 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:41:15.034396 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:15.034361 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h"] Nov 02 14:41:15.047309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:15.047286 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cd8266-47c5h"] Nov 02 14:41:15.086852 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:15.086824 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a9bfd5-53c0-4a58-ad92-57d03f6d2710" path="/var/lib/kubelet/pods/a9a9bfd5-53c0-4a58-ad92-57d03f6d2710/volumes" Nov 02 14:41:18.662269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:18.662235 4431 scope.go:122] "RemoveContainer" containerID="c8377756965d23bc14a09e8fe91f59494353ab3649100043013abbf3702a487b" Nov 02 14:41:18.672019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:18.671999 4431 scope.go:122] "RemoveContainer" containerID="f8cba83e45358de3705b49f107e2b7fae4ac0ab8a95adea159bcfa806bfdbef2" Nov 02 14:41:18.682299 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:18.682280 4431 scope.go:122] "RemoveContainer" containerID="c80f98f327a8b375faa265045625f77531d695574718ef236bec3ee09a10a27e" Nov 02 14:41:18.690309 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:18.690289 4431 scope.go:122] "RemoveContainer" containerID="a702ff0eaeae29d927c099cf5e03c50fc4c835458ce4a46ab45c4282fa4ddc96" Nov 02 14:41:18.698426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:18.698406 4431 scope.go:122] "RemoveContainer" containerID="1b782092e82e7c7d6376b546889b0d97783b6cd803cfcb33c061691e0b9e9806" Nov 02 14:41:21.284926 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:41:21.284882 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:41:26.034324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:26.034290 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7"] Nov 02 14:41:26.047465 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:26.047434 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f6f3ec1-j2nf7"] Nov 02 14:41:27.075457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:27.075422 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e48e3a-374f-45e1-aac5-f26c9b89c55c" path="/var/lib/kubelet/pods/01e48e3a-374f-45e1-aac5-f26c9b89c55c/volumes" Nov 02 14:41:28.042147 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:28.042078 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94"] Nov 02 14:41:28.055016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:28.054981 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6fb08d5-csn94"] Nov 02 14:41:29.075128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:29.075093 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2875bf-aabf-4099-b0be-390554272184" path="/var/lib/kubelet/pods/1f2875bf-aabf-4099-b0be-390554272184/volumes" Nov 02 14:41:35.033622 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:35.033591 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz"] Nov 02 14:41:35.046632 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:35.046605 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b73c02-px6qz"] Nov 02 14:41:35.078506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:35.078473 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d52c50-0249-423b-a18c-870855ddca19" path="/var/lib/kubelet/pods/a0d52c50-0249-423b-a18c-870855ddca19/volumes" Nov 02 14:41:37.047209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:37.047176 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb"] Nov 02 14:41:37.058615 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:37.058586 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-951f9f12-thdlb"] Nov 02 14:41:37.075440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:37.075403 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd" path="/var/lib/kubelet/pods/fdc54af7-a3a8-4d2d-9033-a7a6da6f58cd/volumes" Nov 02 14:41:40.933970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:40.933936 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x"] Nov 02 14:41:40.934356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:40.934221 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" containerID="cri-o://d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e" gracePeriod=30 Nov 02 14:41:41.040382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:41.040329 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn"] Nov 02 14:41:41.047426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:41.047387 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-213987ab-bddsn"] Nov 02 14:41:41.075227 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:41.075191 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e" path="/var/lib/kubelet/pods/b6eb92ed-6be3-4f1b-b4ae-ef761ad3b26e/volumes" Nov 02 14:41:42.096240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:42.096205 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:41:43.286352 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:43.286311 4431 patch_prober.go:29] interesting pod/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 14:41:43.286869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:43.286388 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:41:46.196463 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.196440 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:41:46.266248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.266179 4431 generic.go:356] "Generic (PLEG): container finished" podID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerID="d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e" exitCode=0 Nov 02 14:41:46.266248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.266215 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" event={"ID":"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0","Type":"ContainerDied","Data":"d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e"} Nov 02 14:41:46.266248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.266241 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" Nov 02 14:41:46.266497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.266286 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x" event={"ID":"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0","Type":"ContainerDied","Data":"2e03c5013f8d6bc1049d5af25ea272056ab611c5e28e77817879ae14b81f2686"} Nov 02 14:41:46.266497 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.266316 4431 scope.go:122] "RemoveContainer" containerID="d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e" Nov 02 14:41:46.276694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.276675 4431 scope.go:122] "RemoveContainer" containerID="da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba" Nov 02 14:41:46.286128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286105 4431 scope.go:122] "RemoveContainer" containerID="d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e" Nov 02 14:41:46.286327 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286302 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-kube-api-access-c49wf\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-kube-api-access-c49wf\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.286426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286344 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-token\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-token\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.286426 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286381 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-credential-socket\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.286550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286449 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istiod-ca-cert\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.286550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286487 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-socket\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.286671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286547 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-data\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.286671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286637 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-envoy\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.286819 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286752 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-credential-socket" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:46.287035 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.286969 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-socket" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:46.287284 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287260 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-data" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:46.287370 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287299 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istiod-ca-cert" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:41:46.287440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287369 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-certs\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.287440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287434 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-podinfo\") pod \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\" (UID: \"e486b7e9-6294-4ff0-b79e-522f2fbdc4f0\") " Nov 02 14:41:46.288068 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:41:46.287822 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e\": container with ID starting with d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e not found: ID does not exist" containerID="d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e" Nov 02 14:41:46.288068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287871 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.288068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287892 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.288068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287907 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.288068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287919 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.288068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287866 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e"} err="failed to get container status \"d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e\": rpc error: code = NotFound desc = could not find container \"d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e\": container with ID starting with d625075defcc8e6c7c7668ccec4c31931db2d09bf5fbc50296111f84199a4c3e not found: ID does not exist" Nov 02 14:41:46.288068 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.287938 4431 scope.go:122] "RemoveContainer" containerID="da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba" Nov 02 14:41:46.288589 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.288553 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-certs" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:46.288788 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:41:46.288744 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba\": container with ID starting with da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba not found: ID does not exist" containerID="da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba" Nov 02 14:41:46.288886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.288811 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba"} err="failed to get container status \"da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba\": rpc error: code = NotFound desc = could not find container \"da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba\": container with ID starting with da5b51f9f993698d61ee39b7630a4152169ccc65f76c2fd699926a90fee6f8ba not found: ID does not exist" Nov 02 14:41:46.289567 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.289543 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-token" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:46.289672 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.289630 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-envoy" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:41:46.289893 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.289873 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-kube-api-access-c49wf" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "kube-api-access-c49wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:41:46.290845 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.290822 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-podinfo" pod "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" (UID: "e486b7e9-6294-4ff0-b79e-522f2fbdc4f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:41:46.388546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.388522 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.388546 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.388544 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.388680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.388555 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.388680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.388564 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c49wf\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-kube-api-access-c49wf\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.388680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.388572 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:41:46.616601 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.616579 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x"] Nov 02 14:41:46.630421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:46.630397 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-944506-openshift-default-855597c485-fdk9x"] Nov 02 14:41:47.075364 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:47.075321 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" path="/var/lib/kubelet/pods/e486b7e9-6294-4ff0-b79e-522f2fbdc4f0/volumes" Nov 02 14:41:48.035246 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:48.035221 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6"] Nov 02 14:41:48.047882 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:48.047855 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9f6d62-tdmb6"] Nov 02 14:41:49.075331 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:49.075297 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a2a004-da83-4244-8f22-53d3eef3c29d" path="/var/lib/kubelet/pods/d2a2a004-da83-4244-8f22-53d3eef3c29d/volumes" Nov 02 14:41:51.291792 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:41:51.291739 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:41:52.491680 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:52.491644 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:41:56.496498 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:56.496463 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn"] Nov 02 14:41:56.500146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:56.500111 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" containerID="cri-o://b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96" gracePeriod=30 Nov 02 14:41:57.460736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.460707 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4"] Nov 02 14:41:57.461160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461147 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" Nov 02 14:41:57.461214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461161 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" Nov 02 14:41:57.461214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461183 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" Nov 02 14:41:57.461214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461188 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" Nov 02 14:41:57.461214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461197 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" Nov 02 14:41:57.461214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461202 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461215 4431 cpu_manager.go:414] "containerMap: removing container" podUID="0b585214-c2fc-4bb8-b1d7-54e12b0667be" containerName="revoke-keys" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461221 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b585214-c2fc-4bb8-b1d7-54e12b0667be" containerName="revoke-keys" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461231 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461236 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461244 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461248 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461254 4431 cpu_manager.go:414] "containerMap: removing container" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461258 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461321 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5edc2bd-c7dd-40b0-9e46-db8efd1428b4" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461330 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461339 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b585214-c2fc-4bb8-b1d7-54e12b0667be" containerName="revoke-keys" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461347 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461352 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43d8539-443f-42d1-b09f-7ec8714c1137" containerName="main" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461356 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb08832-f1bd-44e1-bf69-b9d3a298980b" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461364 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" Nov 02 14:41:57.461375 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.461372 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28d6443-555d-4b37-b286-ccd55a957b01" containerName="main" Nov 02 14:41:57.466204 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.466184 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.486565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.486545 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4"] Nov 02 14:41:57.494432 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.494411 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmklw\" (UniqueName: \"kubernetes.io/projected/7867a9ba-77eb-4417-951d-1b1405c4c30c-kube-api-access-hmklw\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.494564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.494535 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.494657 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.494607 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.496238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.496200 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:41:57.496369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.496241 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:41:57.595739 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.595703 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.596241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.595796 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.596241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.595835 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmklw\" (UniqueName: \"kubernetes.io/projected/7867a9ba-77eb-4417-951d-1b1405c4c30c-kube-api-access-hmklw\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.596575 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.596549 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.598085 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.598062 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.627209 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.627180 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmklw\" (UniqueName: \"kubernetes.io/projected/7867a9ba-77eb-4417-951d-1b1405c4c30c-kube-api-access-hmklw\") pod \"maas-api-revoke-sub-cac4fb65-qpzc4\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:57.779061 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:57.778997 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:41:58.094532 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:58.094511 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4"] Nov 02 14:41:58.096796 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:41:58.096752 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7867a9ba_77eb_4417_951d_1b1405c4c30c.slice/crio-56156253f54bea870c9891935a22489b8c840c06ecaf3d4f952ae87b8427b3fe WatchSource:0}: Error finding container 56156253f54bea870c9891935a22489b8c840c06ecaf3d4f952ae87b8427b3fe: Status 404 returned error can't find the container with id 56156253f54bea870c9891935a22489b8c840c06ecaf3d4f952ae87b8427b3fe Nov 02 14:41:58.348582 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:58.348516 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" event={"ID":"7867a9ba-77eb-4417-951d-1b1405c4c30c","Type":"ContainerStarted","Data":"56156253f54bea870c9891935a22489b8c840c06ecaf3d4f952ae87b8427b3fe"} Nov 02 14:41:59.355757 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:59.355717 4431 generic.go:356] "Generic (PLEG): container finished" podID="7867a9ba-77eb-4417-951d-1b1405c4c30c" containerID="3c4d8712a8aa7e7134496c99859a8482822d2bdd14674c9967fefb9cbed081f7" exitCode=0 Nov 02 14:41:59.356199 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:41:59.355807 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" event={"ID":"7867a9ba-77eb-4417-951d-1b1405c4c30c","Type":"ContainerDied","Data":"3c4d8712a8aa7e7134496c99859a8482822d2bdd14674c9967fefb9cbed081f7"} Nov 02 14:42:00.502034 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.502011 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:42:00.626125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.626043 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-openshift-ca\") pod \"7867a9ba-77eb-4417-951d-1b1405c4c30c\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " Nov 02 14:42:00.626301 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.626185 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-xks-ca\") pod \"7867a9ba-77eb-4417-951d-1b1405c4c30c\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " Nov 02 14:42:00.626379 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.626319 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7867a9ba-77eb-4417-951d-1b1405c4c30c-kube-api-access-hmklw\" (UniqueName: \"kubernetes.io/projected/7867a9ba-77eb-4417-951d-1b1405c4c30c-kube-api-access-hmklw\") pod \"7867a9ba-77eb-4417-951d-1b1405c4c30c\" (UID: \"7867a9ba-77eb-4417-951d-1b1405c4c30c\") " Nov 02 14:42:00.626445 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.626406 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-openshift-ca" pod "7867a9ba-77eb-4417-951d-1b1405c4c30c" (UID: "7867a9ba-77eb-4417-951d-1b1405c4c30c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:00.626654 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.626636 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:00.628344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.628320 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-xks-ca" pod "7867a9ba-77eb-4417-951d-1b1405c4c30c" (UID: "7867a9ba-77eb-4417-951d-1b1405c4c30c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:00.628455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.628404 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7867a9ba-77eb-4417-951d-1b1405c4c30c-kube-api-access-hmklw" pod "7867a9ba-77eb-4417-951d-1b1405c4c30c" (UID: "7867a9ba-77eb-4417-951d-1b1405c4c30c"). InnerVolumeSpecName "kube-api-access-hmklw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:00.727957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.727928 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmklw\" (UniqueName: \"kubernetes.io/projected/7867a9ba-77eb-4417-951d-1b1405c4c30c-kube-api-access-hmklw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:00.727957 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:00.727956 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7867a9ba-77eb-4417-951d-1b1405c4c30c-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.367455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.367387 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" Nov 02 14:42:01.367455 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.367430 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4" event={"ID":"7867a9ba-77eb-4417-951d-1b1405c4c30c","Type":"ContainerDied","Data":"56156253f54bea870c9891935a22489b8c840c06ecaf3d4f952ae87b8427b3fe"} Nov 02 14:42:01.367637 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.367475 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56156253f54bea870c9891935a22489b8c840c06ecaf3d4f952ae87b8427b3fe" Nov 02 14:42:01.751827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.751805 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:42:01.839525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.839490 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-kube-api-access-tj4xw\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-kube-api-access-tj4xw\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.839708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.839547 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-credential-socket\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.839708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.839580 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istiod-ca-cert\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.839708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.839641 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-token\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-token\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.839708 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.839687 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-certs\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.839969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.839720 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-socket\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.839969 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.839942 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-podinfo\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.840079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.839984 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-credential-socket" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:01.840079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840008 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-data\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.840079 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840069 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-socket" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:01.840236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840077 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-envoy\") pod \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\" (UID: \"80d74219-9bc6-492c-b1fb-01a2c2b22ebf\") " Nov 02 14:42:01.840236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840090 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-certs" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:01.840236 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840118 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istiod-ca-cert" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:01.840399 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840262 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-data" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:01.840487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840470 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.840549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840493 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.840549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840507 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.840549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840521 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.840549 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.840534 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.841813 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.841768 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-kube-api-access-tj4xw" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "kube-api-access-tj4xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:01.841973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.841958 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-podinfo" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:42:01.842281 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.842266 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-token" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:01.842344 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.842331 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-envoy" pod "80d74219-9bc6-492c-b1fb-01a2c2b22ebf" (UID: "80d74219-9bc6-492c-b1fb-01a2c2b22ebf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:01.941390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.941348 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.941390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.941382 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.941390 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.941391 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj4xw\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-kube-api-access-tj4xw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.941652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.941403 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d74219-9bc6-492c-b1fb-01a2c2b22ebf-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:01.969762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.969726 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888"] Nov 02 14:42:01.975913 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:01.975887 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh"] Nov 02 14:42:02.376171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.376070 4431 generic.go:356] "Generic (PLEG): container finished" podID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerID="b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96" exitCode=0 Nov 02 14:42:02.376171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.376137 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" Nov 02 14:42:02.376171 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.376144 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" event={"ID":"80d74219-9bc6-492c-b1fb-01a2c2b22ebf","Type":"ContainerDied","Data":"b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96"} Nov 02 14:42:02.376468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.376212 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn" event={"ID":"80d74219-9bc6-492c-b1fb-01a2c2b22ebf","Type":"ContainerDied","Data":"625be102d779e91d65c5695eadb5436d3b3ce9e5b2d39efd49298b3c82cb11b6"} Nov 02 14:42:02.376468 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.376236 4431 scope.go:122] "RemoveContainer" containerID="b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96" Nov 02 14:42:02.392124 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.392094 4431 scope.go:122] "RemoveContainer" containerID="665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212" Nov 02 14:42:02.401588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.401567 4431 scope.go:122] "RemoveContainer" containerID="b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96" Nov 02 14:42:02.401883 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:02.401861 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96\": container with ID starting with b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96 not found: ID does not exist" containerID="b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96" Nov 02 14:42:02.401980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.401892 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96"} err="failed to get container status \"b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96\": rpc error: code = NotFound desc = could not find container \"b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96\": container with ID starting with b9654e597a369ea58c7f81b2f35f9dd4f3b9ea8c24cb886ce3ce3f1967828c96 not found: ID does not exist" Nov 02 14:42:02.401980 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.401909 4431 scope.go:122] "RemoveContainer" containerID="665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212" Nov 02 14:42:02.402186 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:02.402164 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212\": container with ID starting with 665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212 not found: ID does not exist" containerID="665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212" Nov 02 14:42:02.402232 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.402192 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212"} err="failed to get container status \"665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212\": rpc error: code = NotFound desc = could not find container \"665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212\": container with ID starting with 665378c43fc3c03b5730fa085577038f524afde1833f1d7ca739ff479bdb1212 not found: ID does not exist" Nov 02 14:42:02.442812 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.442766 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn"] Nov 02 14:42:02.454800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:02.454746 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-673ba8-openshift-default-755484574f-x9zcn"] Nov 02 14:42:03.076807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.076760 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" path="/var/lib/kubelet/pods/80d74219-9bc6-492c-b1fb-01a2c2b22ebf/volumes" Nov 02 14:42:03.271864 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.271836 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p"] Nov 02 14:42:03.272268 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272255 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" Nov 02 14:42:03.272316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272269 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e486b7e9-6294-4ff0-b79e-522f2fbdc4f0" containerName="istio-proxy" Nov 02 14:42:03.272316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272301 4431 cpu_manager.go:414] "containerMap: removing container" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" Nov 02 14:42:03.272316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272306 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" Nov 02 14:42:03.272413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272316 4431 cpu_manager.go:414] "containerMap: removing container" podUID="7867a9ba-77eb-4417-951d-1b1405c4c30c" containerName="revoke-keys" Nov 02 14:42:03.272413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272321 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7867a9ba-77eb-4417-951d-1b1405c4c30c" containerName="revoke-keys" Nov 02 14:42:03.272413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272390 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" Nov 02 14:42:03.272413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272399 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="7867a9ba-77eb-4417-951d-1b1405c4c30c" containerName="revoke-keys" Nov 02 14:42:03.272413 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.272407 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" Nov 02 14:42:03.277112 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.277091 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.307533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.307510 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:42:03.307649 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.307577 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:42:03.310125 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.310108 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p"] Nov 02 14:42:03.355408 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.355352 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtd4\" (UniqueName: \"kubernetes.io/projected/faf3f0a9-c255-485a-8aa4-c820eed9630a-kube-api-access-fgtd4\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.355506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.355469 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.355570 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.355510 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.456804 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.456753 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtd4\" (UniqueName: \"kubernetes.io/projected/faf3f0a9-c255-485a-8aa4-c820eed9630a-kube-api-access-fgtd4\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.456934 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.456889 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.457004 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.456933 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.457523 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.457500 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.459284 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.459263 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.481872 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.481847 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtd4\" (UniqueName: \"kubernetes.io/projected/faf3f0a9-c255-485a-8aa4-c820eed9630a-kube-api-access-fgtd4\") pod \"maas-api-revoke-sub-8fad2bfd-pj67p\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.590137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.590116 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:03.892159 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:03.892133 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p"] Nov 02 14:42:03.893564 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:42:03.893525 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf3f0a9_c255_485a_8aa4_c820eed9630a.slice/crio-a1971a09b8e5b9bf35b8e74f42a5082900532070491139e928ec6e51b6e81d34 WatchSource:0}: Error finding container a1971a09b8e5b9bf35b8e74f42a5082900532070491139e928ec6e51b6e81d34: Status 404 returned error can't find the container with id a1971a09b8e5b9bf35b8e74f42a5082900532070491139e928ec6e51b6e81d34 Nov 02 14:42:04.390382 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:04.390345 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" event={"ID":"faf3f0a9-c255-485a-8aa4-c820eed9630a","Type":"ContainerStarted","Data":"e4285949c8f125c854e809c84bd9497f8a142fba29a31f559e2583df4c9f866b"} Nov 02 14:42:04.390821 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:04.390406 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" event={"ID":"faf3f0a9-c255-485a-8aa4-c820eed9630a","Type":"ContainerStarted","Data":"a1971a09b8e5b9bf35b8e74f42a5082900532070491139e928ec6e51b6e81d34"} Nov 02 14:42:04.447221 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:04.447137 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" podStartSLOduration=1.447124457 podStartE2EDuration="1.447124457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:42:04.444823778 +0000 UTC m=+2689.481928711" watchObservedRunningTime="2026-11-02 14:42:04.447124457 +0000 UTC m=+2689.484229389" Nov 02 14:42:05.398073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:05.398040 4431 generic.go:356] "Generic (PLEG): container finished" podID="faf3f0a9-c255-485a-8aa4-c820eed9630a" containerID="e4285949c8f125c854e809c84bd9497f8a142fba29a31f559e2583df4c9f866b" exitCode=0 Nov 02 14:42:05.398544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:05.398122 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" event={"ID":"faf3f0a9-c255-485a-8aa4-c820eed9630a","Type":"ContainerDied","Data":"e4285949c8f125c854e809c84bd9497f8a142fba29a31f559e2583df4c9f866b"} Nov 02 14:42:06.531073 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.531050 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:06.591728 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.591693 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-openshift-ca\") pod \"faf3f0a9-c255-485a-8aa4-c820eed9630a\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " Nov 02 14:42:06.591968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.591931 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-xks-ca\") pod \"faf3f0a9-c255-485a-8aa4-c820eed9630a\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " Nov 02 14:42:06.592114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.592029 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf3f0a9-c255-485a-8aa4-c820eed9630a-kube-api-access-fgtd4\" (UniqueName: \"kubernetes.io/projected/faf3f0a9-c255-485a-8aa4-c820eed9630a-kube-api-access-fgtd4\") pod \"faf3f0a9-c255-485a-8aa4-c820eed9630a\" (UID: \"faf3f0a9-c255-485a-8aa4-c820eed9630a\") " Nov 02 14:42:06.592240 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.592122 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-openshift-ca" pod "faf3f0a9-c255-485a-8aa4-c820eed9630a" (UID: "faf3f0a9-c255-485a-8aa4-c820eed9630a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:06.592663 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.592638 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:06.594621 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.594597 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf3f0a9-c255-485a-8aa4-c820eed9630a-kube-api-access-fgtd4" pod "faf3f0a9-c255-485a-8aa4-c820eed9630a" (UID: "faf3f0a9-c255-485a-8aa4-c820eed9630a"). InnerVolumeSpecName "kube-api-access-fgtd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:06.594736 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.594679 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-xks-ca" pod "faf3f0a9-c255-485a-8aa4-c820eed9630a" (UID: "faf3f0a9-c255-485a-8aa4-c820eed9630a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:06.693678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.693610 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faf3f0a9-c255-485a-8aa4-c820eed9630a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:06.693678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.693658 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtd4\" (UniqueName: \"kubernetes.io/projected/faf3f0a9-c255-485a-8aa4-c820eed9630a-kube-api-access-fgtd4\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:06.905398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:06.905367 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:42:07.411210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:07.411184 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" Nov 02 14:42:07.411403 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:07.411214 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p" event={"ID":"faf3f0a9-c255-485a-8aa4-c820eed9630a","Type":"ContainerDied","Data":"a1971a09b8e5b9bf35b8e74f42a5082900532070491139e928ec6e51b6e81d34"} Nov 02 14:42:07.411403 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:07.411262 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1971a09b8e5b9bf35b8e74f42a5082900532070491139e928ec6e51b6e81d34" Nov 02 14:42:07.784965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:07.784883 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w"] Nov 02 14:42:07.797381 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:07.797349 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4"] Nov 02 14:42:09.289850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.289820 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w"] Nov 02 14:42:09.290423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.290405 4431 cpu_manager.go:414] "containerMap: removing container" podUID="faf3f0a9-c255-485a-8aa4-c820eed9630a" containerName="revoke-keys" Nov 02 14:42:09.290423 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.290424 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf3f0a9-c255-485a-8aa4-c820eed9630a" containerName="revoke-keys" Nov 02 14:42:09.290527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.290453 4431 cpu_manager.go:414] "containerMap: removing container" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" Nov 02 14:42:09.290527 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.290459 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d74219-9bc6-492c-b1fb-01a2c2b22ebf" containerName="istio-proxy" Nov 02 14:42:09.290610 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.290536 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf3f0a9-c255-485a-8aa4-c820eed9630a" containerName="revoke-keys" Nov 02 14:42:09.293768 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.293749 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.321668 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.321648 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w"] Nov 02 14:42:09.331383 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.331355 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:42:09.331544 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.331527 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:42:09.422151 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.422124 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs8dr\" (UniqueName: \"kubernetes.io/projected/977f48d8-3f65-4bb4-82cf-a23f46a3740a-kube-api-access-qs8dr\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.422279 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.422212 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.422334 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.422325 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.523195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.523144 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.523195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.523190 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8dr\" (UniqueName: \"kubernetes.io/projected/977f48d8-3f65-4bb4-82cf-a23f46a3740a-kube-api-access-qs8dr\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.523590 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.523238 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.523843 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.523821 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.525528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.525505 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.548561 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.548491 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8dr\" (UniqueName: \"kubernetes.io/projected/977f48d8-3f65-4bb4-82cf-a23f46a3740a-kube-api-access-qs8dr\") pod \"maas-api-revoke-sub-4dfe8972-kqx9w\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.607218 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.607182 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:09.913635 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:09.913610 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w"] Nov 02 14:42:09.915071 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:42:09.915046 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977f48d8_3f65_4bb4_82cf_a23f46a3740a.slice/crio-0a9feb0a5aa7f9b75280cf498a0db79d1fd7e3737606ce9b6c1b6aa89b8ecca6 WatchSource:0}: Error finding container 0a9feb0a5aa7f9b75280cf498a0db79d1fd7e3737606ce9b6c1b6aa89b8ecca6: Status 404 returned error can't find the container with id 0a9feb0a5aa7f9b75280cf498a0db79d1fd7e3737606ce9b6c1b6aa89b8ecca6 Nov 02 14:42:10.431891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:10.431809 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" event={"ID":"977f48d8-3f65-4bb4-82cf-a23f46a3740a","Type":"ContainerStarted","Data":"5be065b7647c6b97e30107ce455d2198e1109e7ed9ef6c50a192fab7be2a11c6"} Nov 02 14:42:10.431891 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:10.431863 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" event={"ID":"977f48d8-3f65-4bb4-82cf-a23f46a3740a","Type":"ContainerStarted","Data":"0a9feb0a5aa7f9b75280cf498a0db79d1fd7e3737606ce9b6c1b6aa89b8ecca6"} Nov 02 14:42:10.492524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:10.492475 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" podStartSLOduration=1.49246225 podStartE2EDuration="1.49246225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:42:10.49087412 +0000 UTC m=+2695.527979054" watchObservedRunningTime="2026-11-02 14:42:10.49246225 +0000 UTC m=+2695.529567223" Nov 02 14:42:11.439927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:11.439882 4431 generic.go:356] "Generic (PLEG): container finished" podID="977f48d8-3f65-4bb4-82cf-a23f46a3740a" containerID="5be065b7647c6b97e30107ce455d2198e1109e7ed9ef6c50a192fab7be2a11c6" exitCode=0 Nov 02 14:42:11.439927 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:11.439921 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" event={"ID":"977f48d8-3f65-4bb4-82cf-a23f46a3740a","Type":"ContainerDied","Data":"5be065b7647c6b97e30107ce455d2198e1109e7ed9ef6c50a192fab7be2a11c6"} Nov 02 14:42:12.613931 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.613885 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:12.754328 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.754246 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977f48d8-3f65-4bb4-82cf-a23f46a3740a-kube-api-access-qs8dr\" (UniqueName: \"kubernetes.io/projected/977f48d8-3f65-4bb4-82cf-a23f46a3740a-kube-api-access-qs8dr\") pod \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " Nov 02 14:42:12.754508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.754331 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-xks-ca\") pod \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " Nov 02 14:42:12.754508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.754382 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-openshift-ca\") pod \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\" (UID: \"977f48d8-3f65-4bb4-82cf-a23f46a3740a\") " Nov 02 14:42:12.754807 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.754764 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-openshift-ca" pod "977f48d8-3f65-4bb4-82cf-a23f46a3740a" (UID: "977f48d8-3f65-4bb4-82cf-a23f46a3740a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:12.756451 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.756423 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977f48d8-3f65-4bb4-82cf-a23f46a3740a-kube-api-access-qs8dr" pod "977f48d8-3f65-4bb4-82cf-a23f46a3740a" (UID: "977f48d8-3f65-4bb4-82cf-a23f46a3740a"). InnerVolumeSpecName "kube-api-access-qs8dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:12.756560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.756504 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-xks-ca" pod "977f48d8-3f65-4bb4-82cf-a23f46a3740a" (UID: "977f48d8-3f65-4bb4-82cf-a23f46a3740a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:12.855494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.855463 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs8dr\" (UniqueName: \"kubernetes.io/projected/977f48d8-3f65-4bb4-82cf-a23f46a3740a-kube-api-access-qs8dr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:12.855494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.855489 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:12.855494 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:12.855499 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/977f48d8-3f65-4bb4-82cf-a23f46a3740a-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:13.453748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.453717 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" event={"ID":"977f48d8-3f65-4bb4-82cf-a23f46a3740a","Type":"ContainerDied","Data":"0a9feb0a5aa7f9b75280cf498a0db79d1fd7e3737606ce9b6c1b6aa89b8ecca6"} Nov 02 14:42:13.453914 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.453765 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9feb0a5aa7f9b75280cf498a0db79d1fd7e3737606ce9b6c1b6aa89b8ecca6" Nov 02 14:42:13.453914 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.453732 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w" Nov 02 14:42:13.734651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.734568 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp"] Nov 02 14:42:13.735097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.735028 4431 cpu_manager.go:414] "containerMap: removing container" podUID="977f48d8-3f65-4bb4-82cf-a23f46a3740a" containerName="revoke-keys" Nov 02 14:42:13.735097 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.735040 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977f48d8-3f65-4bb4-82cf-a23f46a3740a" containerName="revoke-keys" Nov 02 14:42:13.735174 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.735124 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="977f48d8-3f65-4bb4-82cf-a23f46a3740a" containerName="revoke-keys" Nov 02 14:42:13.738528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.738509 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.757210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.757185 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:42:13.757357 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.757344 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n6q\"" Nov 02 14:42:13.760148 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.760126 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp"] Nov 02 14:42:13.865300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.865274 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.865300 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.865305 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.865534 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.865445 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppqd\" (UniqueName: \"kubernetes.io/projected/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-kube-api-access-5ppqd\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.966316 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.966283 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppqd\" (UniqueName: \"kubernetes.io/projected/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-kube-api-access-5ppqd\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.966463 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.966444 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.966504 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.966480 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.967067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.967050 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.968748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.968730 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:13.991294 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:13.991232 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppqd\" (UniqueName: \"kubernetes.io/projected/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-kube-api-access-5ppqd\") pod \"maas-api-revoke-sub-566535a3-lrmfp\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:14.051271 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:14.051241 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:14.366482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:14.366451 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp"] Nov 02 14:42:14.369547 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:42:14.368547 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba931fbf_a2a2_4d1e_bab0_bc555b3280a0.slice/crio-cf55044cf159a2fe10f5f72efa97e1084c85c5b0c73e33e144ddafa62a5883aa WatchSource:0}: Error finding container cf55044cf159a2fe10f5f72efa97e1084c85c5b0c73e33e144ddafa62a5883aa: Status 404 returned error can't find the container with id cf55044cf159a2fe10f5f72efa97e1084c85c5b0c73e33e144ddafa62a5883aa Nov 02 14:42:14.461247 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:14.461220 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" event={"ID":"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0","Type":"ContainerStarted","Data":"cf55044cf159a2fe10f5f72efa97e1084c85c5b0c73e33e144ddafa62a5883aa"} Nov 02 14:42:15.468702 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:15.468623 4431 generic.go:356] "Generic (PLEG): container finished" podID="ba931fbf-a2a2-4d1e-bab0-bc555b3280a0" containerID="3df2d7e3828e92ad7ce823a4f6f6d714b3ec4b432795af722be78dd37ffc39ce" exitCode=0 Nov 02 14:42:15.469093 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:15.468711 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" event={"ID":"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0","Type":"ContainerDied","Data":"3df2d7e3828e92ad7ce823a4f6f6d714b3ec4b432795af722be78dd37ffc39ce"} Nov 02 14:42:16.603531 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.603505 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:16.694477 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.694453 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-xks-ca\") pod \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " Nov 02 14:42:16.694608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.694523 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-kube-api-access-5ppqd\" (UniqueName: \"kubernetes.io/projected/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-kube-api-access-5ppqd\") pod \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " Nov 02 14:42:16.694608 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.694568 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-openshift-ca\") pod \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\" (UID: \"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0\") " Nov 02 14:42:16.695025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.694997 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-openshift-ca" pod "ba931fbf-a2a2-4d1e-bab0-bc555b3280a0" (UID: "ba931fbf-a2a2-4d1e-bab0-bc555b3280a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:16.696553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.696524 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-kube-api-access-5ppqd" pod "ba931fbf-a2a2-4d1e-bab0-bc555b3280a0" (UID: "ba931fbf-a2a2-4d1e-bab0-bc555b3280a0"). InnerVolumeSpecName "kube-api-access-5ppqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:16.696553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.696525 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-xks-ca" pod "ba931fbf-a2a2-4d1e-bab0-bc555b3280a0" (UID: "ba931fbf-a2a2-4d1e-bab0-bc555b3280a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:16.795372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.795317 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:16.795372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.795341 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ppqd\" (UniqueName: \"kubernetes.io/projected/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-kube-api-access-5ppqd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:16.795372 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.795356 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:16.994521 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.994464 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" containerID="cri-o://5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2" gracePeriod=45 Nov 02 14:42:16.998851 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:16.998794 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" containerID="cri-o://f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55" gracePeriod=45 Nov 02 14:42:17.239479 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.239452 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:42:17.276960 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.276931 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:42:17.401530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401498 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-dshm\") pod \"2d005e58-763a-4c16-98d9-fea871db9498\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " Nov 02 14:42:17.401705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401549 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-kube-api-access-t2b2s\" (UniqueName: \"kubernetes.io/projected/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-kube-api-access-t2b2s\") pod \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " Nov 02 14:42:17.401705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401572 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-dshm\") pod \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " Nov 02 14:42:17.401705 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401667 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tmp-dir\") pod \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " Nov 02 14:42:17.401923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401756 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d005e58-763a-4c16-98d9-fea871db9498-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d005e58-763a-4c16-98d9-fea871db9498-tls-certs\") pod \"2d005e58-763a-4c16-98d9-fea871db9498\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " Nov 02 14:42:17.401923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401802 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-home\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-home\") pod \"2d005e58-763a-4c16-98d9-fea871db9498\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " Nov 02 14:42:17.401923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401870 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d005e58-763a-4c16-98d9-fea871db9498-kube-api-access-t4fs5\" (UniqueName: \"kubernetes.io/projected/2d005e58-763a-4c16-98d9-fea871db9498-kube-api-access-t4fs5\") pod \"2d005e58-763a-4c16-98d9-fea871db9498\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " Nov 02 14:42:17.401923 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401902 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-home\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-home\") pod \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " Nov 02 14:42:17.402152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401937 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-model-cache\") pod \"2d005e58-763a-4c16-98d9-fea871db9498\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " Nov 02 14:42:17.402152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.401967 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tmp-dir" pod "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" (UID: "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:17.402152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402012 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-tmp-dir\") pod \"2d005e58-763a-4c16-98d9-fea871db9498\" (UID: \"2d005e58-763a-4c16-98d9-fea871db9498\") " Nov 02 14:42:17.402152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402111 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tls-certs\") pod \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " Nov 02 14:42:17.402152 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402143 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-model-cache\") pod \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\" (UID: \"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c\") " Nov 02 14:42:17.402421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402319 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-model-cache" pod "2d005e58-763a-4c16-98d9-fea871db9498" (UID: "2d005e58-763a-4c16-98d9-fea871db9498"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:17.402421 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402340 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-home" pod "2d005e58-763a-4c16-98d9-fea871db9498" (UID: "2d005e58-763a-4c16-98d9-fea871db9498"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:17.402543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402513 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-model-cache" pod "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" (UID: "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:17.402543 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402528 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-home" pod "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" (UID: "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:17.402651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402627 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.402651 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402646 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.402825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402681 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.402825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402694 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.402825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402707 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.402825 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.402725 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-tmp-dir" pod "2d005e58-763a-4c16-98d9-fea871db9498" (UID: "2d005e58-763a-4c16-98d9-fea871db9498"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:17.404198 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.404169 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-dshm" pod "2d005e58-763a-4c16-98d9-fea871db9498" (UID: "2d005e58-763a-4c16-98d9-fea871db9498"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:17.404326 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.404246 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-dshm" pod "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" (UID: "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:17.404425 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.404408 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d005e58-763a-4c16-98d9-fea871db9498-kube-api-access-t4fs5" pod "2d005e58-763a-4c16-98d9-fea871db9498" (UID: "2d005e58-763a-4c16-98d9-fea871db9498"). InnerVolumeSpecName "kube-api-access-t4fs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:17.404694 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.404675 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-kube-api-access-t2b2s" pod "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" (UID: "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c"). InnerVolumeSpecName "kube-api-access-t2b2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:17.405086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.405064 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d005e58-763a-4c16-98d9-fea871db9498-tls-certs" pod "2d005e58-763a-4c16-98d9-fea871db9498" (UID: "2d005e58-763a-4c16-98d9-fea871db9498"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:17.405086 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.405075 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tls-certs" pod "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" (UID: "d37e3fc7-7ac2-4b12-90a6-78e8730dc69c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:17.482487 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.482458 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" Nov 02 14:42:17.482671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.482463 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp" event={"ID":"ba931fbf-a2a2-4d1e-bab0-bc555b3280a0","Type":"ContainerDied","Data":"cf55044cf159a2fe10f5f72efa97e1084c85c5b0c73e33e144ddafa62a5883aa"} Nov 02 14:42:17.482671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.482593 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf55044cf159a2fe10f5f72efa97e1084c85c5b0c73e33e144ddafa62a5883aa" Nov 02 14:42:17.484030 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.484006 4431 generic.go:356] "Generic (PLEG): container finished" podID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerID="f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55" exitCode=0 Nov 02 14:42:17.484160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.484048 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" Nov 02 14:42:17.484160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.484086 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" event={"ID":"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c","Type":"ContainerDied","Data":"f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55"} Nov 02 14:42:17.484160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.484131 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh" event={"ID":"d37e3fc7-7ac2-4b12-90a6-78e8730dc69c","Type":"ContainerDied","Data":"af1c9a3d02040d3c271b281dd09600141a533a9a6f1e67738b5a4a7a0b860996"} Nov 02 14:42:17.484160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.484153 4431 scope.go:122] "RemoveContainer" containerID="f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55" Nov 02 14:42:17.486764 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.486742 4431 generic.go:356] "Generic (PLEG): container finished" podID="2d005e58-763a-4c16-98d9-fea871db9498" containerID="5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2" exitCode=0 Nov 02 14:42:17.486946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.486802 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" event={"ID":"2d005e58-763a-4c16-98d9-fea871db9498","Type":"ContainerDied","Data":"5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2"} Nov 02 14:42:17.486946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.486842 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" event={"ID":"2d005e58-763a-4c16-98d9-fea871db9498","Type":"ContainerDied","Data":"57dfc94617329e5159c95f38589539101f2f550345c28ad01de9e6f1a6d71e37"} Nov 02 14:42:17.486946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.486808 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888" Nov 02 14:42:17.496311 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.496296 4431 scope.go:122] "RemoveContainer" containerID="f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55" Nov 02 14:42:17.496630 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:17.496600 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55\": container with ID starting with f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55 not found: ID does not exist" containerID="f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55" Nov 02 14:42:17.496720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.496632 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55"} err="failed to get container status \"f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55\": rpc error: code = NotFound desc = could not find container \"f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55\": container with ID starting with f733cd45d45144961cc369dda8267f3fe97ddea1429af0f362ebcfa0c44e4f55 not found: ID does not exist" Nov 02 14:42:17.496720 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.496654 4431 scope.go:122] "RemoveContainer" containerID="5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2" Nov 02 14:42:17.504150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.504129 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.504150 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.504150 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.504278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.504159 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d005e58-763a-4c16-98d9-fea871db9498-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.504278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.504168 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2b2s\" (UniqueName: \"kubernetes.io/projected/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-kube-api-access-t2b2s\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.504278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.504177 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.504278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.504186 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d005e58-763a-4c16-98d9-fea871db9498-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.504278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.504194 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4fs5\" (UniqueName: \"kubernetes.io/projected/2d005e58-763a-4c16-98d9-fea871db9498-kube-api-access-t4fs5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:17.508163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.508125 4431 scope.go:122] "RemoveContainer" containerID="5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2" Nov 02 14:42:17.508417 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:17.508400 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2\": container with ID starting with 5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2 not found: ID does not exist" containerID="5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2" Nov 02 14:42:17.508474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.508424 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2"} err="failed to get container status \"5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2\": rpc error: code = NotFound desc = could not find container \"5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2\": container with ID starting with 5dada6051bf3b50e1963ddb9680688291c973153b9cf55d2a9f3f5aa32afb8d2 not found: ID does not exist" Nov 02 14:42:17.559183 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.559153 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888"] Nov 02 14:42:17.561584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.561565 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-2-w3-866159-kserve-794b5d6cdd-bn888"] Nov 02 14:42:17.600506 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.600475 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh"] Nov 02 14:42:17.604447 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:17.604423 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-866159/distinct-w3-866159-kserve-5857f4f7f-bxvjh"] Nov 02 14:42:18.746217 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:18.746189 4431 scope.go:122] "RemoveContainer" containerID="b9bfa330a183e0a6744b5a0b0a6ef0bf388a3403fc4425c7b715d0be916442ca" Nov 02 14:42:18.755154 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:18.755135 4431 scope.go:122] "RemoveContainer" containerID="d14563e3836e117a508aedb577602906434c09bc0ba2457f13820228238b3d6e" Nov 02 14:42:19.039261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:19.039187 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:42:19.074949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:19.074916 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d005e58-763a-4c16-98d9-fea871db9498" path="/var/lib/kubelet/pods/2d005e58-763a-4c16-98d9-fea871db9498/volumes" Nov 02 14:42:19.075391 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:19.075377 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" path="/var/lib/kubelet/pods/d37e3fc7-7ac2-4b12-90a6-78e8730dc69c/volumes" Nov 02 14:42:21.282920 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:21.282878 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:42:22.805458 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:22.805408 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" podUID="537d9533-ce0c-4f93-be2f-684bce472885" containerName="main" containerID="cri-o://41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab" gracePeriod=45 Nov 02 14:42:22.816280 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:22.816242 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" podUID="17b062be-55b8-4b08-8512-e8c86b10e317" containerName="main" containerID="cri-o://d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2" gracePeriod=45 Nov 02 14:42:23.069537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.069479 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:42:23.093378 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.093357 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:42:23.157505 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157472 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b062be-55b8-4b08-8512-e8c86b10e317-kube-api-access-d67rr\" (UniqueName: \"kubernetes.io/projected/17b062be-55b8-4b08-8512-e8c86b10e317-kube-api-access-d67rr\") pod \"17b062be-55b8-4b08-8512-e8c86b10e317\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " Nov 02 14:42:23.157678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157556 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-tmp-dir\") pod \"537d9533-ce0c-4f93-be2f-684bce472885\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " Nov 02 14:42:23.157678 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157606 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-home\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-home\") pod \"17b062be-55b8-4b08-8512-e8c86b10e317\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " Nov 02 14:42:23.157814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157696 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-model-cache\") pod \"17b062be-55b8-4b08-8512-e8c86b10e317\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " Nov 02 14:42:23.157814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157724 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-dshm\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-dshm\") pod \"537d9533-ce0c-4f93-be2f-684bce472885\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " Nov 02 14:42:23.157814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157755 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-dshm\") pod \"17b062be-55b8-4b08-8512-e8c86b10e317\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " Nov 02 14:42:23.157814 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157805 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-model-cache\") pod \"537d9533-ce0c-4f93-be2f-684bce472885\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " Nov 02 14:42:23.158043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157817 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-tmp-dir" pod "537d9533-ce0c-4f93-be2f-684bce472885" (UID: "537d9533-ce0c-4f93-be2f-684bce472885"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:23.158043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157836 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-tmp-dir\") pod \"17b062be-55b8-4b08-8512-e8c86b10e317\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " Nov 02 14:42:23.158043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157897 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-home\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-home\") pod \"537d9533-ce0c-4f93-be2f-684bce472885\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " Nov 02 14:42:23.158043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157944 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/537d9533-ce0c-4f93-be2f-684bce472885-kube-api-access-fht9r\" (UniqueName: \"kubernetes.io/projected/537d9533-ce0c-4f93-be2f-684bce472885-kube-api-access-fht9r\") pod \"537d9533-ce0c-4f93-be2f-684bce472885\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " Nov 02 14:42:23.158043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.157977 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/537d9533-ce0c-4f93-be2f-684bce472885-tls-certs\" (UniqueName: \"kubernetes.io/secret/537d9533-ce0c-4f93-be2f-684bce472885-tls-certs\") pod \"537d9533-ce0c-4f93-be2f-684bce472885\" (UID: \"537d9533-ce0c-4f93-be2f-684bce472885\") " Nov 02 14:42:23.158043 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.158009 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17b062be-55b8-4b08-8512-e8c86b10e317-tls-certs\" (UniqueName: \"kubernetes.io/secret/17b062be-55b8-4b08-8512-e8c86b10e317-tls-certs\") pod \"17b062be-55b8-4b08-8512-e8c86b10e317\" (UID: \"17b062be-55b8-4b08-8512-e8c86b10e317\") " Nov 02 14:42:23.158386 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.158363 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-model-cache" pod "537d9533-ce0c-4f93-be2f-684bce472885" (UID: "537d9533-ce0c-4f93-be2f-684bce472885"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:23.158475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.158458 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.158533 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.158481 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.158816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.158791 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-home" pod "537d9533-ce0c-4f93-be2f-684bce472885" (UID: "537d9533-ce0c-4f93-be2f-684bce472885"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:23.159308 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.159032 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-model-cache" pod "17b062be-55b8-4b08-8512-e8c86b10e317" (UID: "17b062be-55b8-4b08-8512-e8c86b10e317"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:23.159511 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.159452 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-tmp-dir" pod "17b062be-55b8-4b08-8512-e8c86b10e317" (UID: "17b062be-55b8-4b08-8512-e8c86b10e317"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:23.159675 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.159647 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-home" pod "17b062be-55b8-4b08-8512-e8c86b10e317" (UID: "17b062be-55b8-4b08-8512-e8c86b10e317"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:23.160365 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.160340 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b062be-55b8-4b08-8512-e8c86b10e317-kube-api-access-d67rr" pod "17b062be-55b8-4b08-8512-e8c86b10e317" (UID: "17b062be-55b8-4b08-8512-e8c86b10e317"). InnerVolumeSpecName "kube-api-access-d67rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:23.160474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.160343 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-dshm" pod "17b062be-55b8-4b08-8512-e8c86b10e317" (UID: "17b062be-55b8-4b08-8512-e8c86b10e317"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:23.160661 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.160634 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-dshm" pod "537d9533-ce0c-4f93-be2f-684bce472885" (UID: "537d9533-ce0c-4f93-be2f-684bce472885"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:23.160853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.160832 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537d9533-ce0c-4f93-be2f-684bce472885-kube-api-access-fht9r" pod "537d9533-ce0c-4f93-be2f-684bce472885" (UID: "537d9533-ce0c-4f93-be2f-684bce472885"). InnerVolumeSpecName "kube-api-access-fht9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:23.161009 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.160991 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b062be-55b8-4b08-8512-e8c86b10e317-tls-certs" pod "17b062be-55b8-4b08-8512-e8c86b10e317" (UID: "17b062be-55b8-4b08-8512-e8c86b10e317"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:23.161712 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.161690 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537d9533-ce0c-4f93-be2f-684bce472885-tls-certs" pod "537d9533-ce0c-4f93-be2f-684bce472885" (UID: "537d9533-ce0c-4f93-be2f-684bce472885"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:42:23.259645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259615 4431 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-model-cache\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259643 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259654 4431 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-dshm\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259665 4431 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-tmp-dir\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259675 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537d9533-ce0c-4f93-be2f-684bce472885-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259688 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fht9r\" (UniqueName: \"kubernetes.io/projected/537d9533-ce0c-4f93-be2f-684bce472885-kube-api-access-fht9r\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259700 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537d9533-ce0c-4f93-be2f-684bce472885-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259712 4431 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b062be-55b8-4b08-8512-e8c86b10e317-tls-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259723 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d67rr\" (UniqueName: \"kubernetes.io/projected/17b062be-55b8-4b08-8512-e8c86b10e317-kube-api-access-d67rr\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.259890 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.259735 4431 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17b062be-55b8-4b08-8512-e8c86b10e317-home\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:23.527180 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.527140 4431 generic.go:356] "Generic (PLEG): container finished" podID="537d9533-ce0c-4f93-be2f-684bce472885" containerID="41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab" exitCode=0 Nov 02 14:42:23.527356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.527188 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" Nov 02 14:42:23.527356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.527225 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" event={"ID":"537d9533-ce0c-4f93-be2f-684bce472885","Type":"ContainerDied","Data":"41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab"} Nov 02 14:42:23.527356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.527271 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w" event={"ID":"537d9533-ce0c-4f93-be2f-684bce472885","Type":"ContainerDied","Data":"09a38aee47073c0d4473e75e4c15ad726f7cecb0bd163ab12c6b35dc84f88e04"} Nov 02 14:42:23.527356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.527290 4431 scope.go:122] "RemoveContainer" containerID="41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab" Nov 02 14:42:23.528850 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.528825 4431 generic.go:356] "Generic (PLEG): container finished" podID="17b062be-55b8-4b08-8512-e8c86b10e317" containerID="d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2" exitCode=0 Nov 02 14:42:23.528982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.528867 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" Nov 02 14:42:23.528982 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.528865 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" event={"ID":"17b062be-55b8-4b08-8512-e8c86b10e317","Type":"ContainerDied","Data":"d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2"} Nov 02 14:42:23.529106 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.528977 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4" event={"ID":"17b062be-55b8-4b08-8512-e8c86b10e317","Type":"ContainerDied","Data":"5e3ef1b848c1ee472fe7ae63171ac09767de3151bb2bf23755f839b3f4cdee53"} Nov 02 14:42:23.537816 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.537797 4431 scope.go:122] "RemoveContainer" containerID="41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab" Nov 02 14:42:23.538097 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:23.538071 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab\": container with ID starting with 41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab not found: ID does not exist" containerID="41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab" Nov 02 14:42:23.538196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.538105 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab"} err="failed to get container status \"41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab\": rpc error: code = NotFound desc = could not find container \"41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab\": container with ID starting with 41b4b8c699499fb9dffe52c56d934b69584c22c5b10dcffeebde64c99154cdab not found: ID does not exist" Nov 02 14:42:23.538196 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.538122 4431 scope.go:122] "RemoveContainer" containerID="d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2" Nov 02 14:42:23.546481 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.546465 4431 scope.go:122] "RemoveContainer" containerID="d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2" Nov 02 14:42:23.546725 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:23.546706 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2\": container with ID starting with d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2 not found: ID does not exist" containerID="d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2" Nov 02 14:42:23.546792 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.546731 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2"} err="failed to get container status \"d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2\": rpc error: code = NotFound desc = could not find container \"d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2\": container with ID starting with d9dbf036c55e9d084f0f5c084c7327b21e34e87dc5e89bf92c1129fa678368d2 not found: ID does not exist" Nov 02 14:42:23.583242 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.583218 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4"] Nov 02 14:42:23.596369 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.596348 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/simulator-w1-f72bc6-kserve-74cf7957b6-ddjb4"] Nov 02 14:42:23.629332 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.629307 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w"] Nov 02 14:42:23.634109 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:23.634088 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f72bc6/premium-w1-f72bc6-kserve-7bd57c9675-wls2w"] Nov 02 14:42:25.074835 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:25.074797 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b062be-55b8-4b08-8512-e8c86b10e317" path="/var/lib/kubelet/pods/17b062be-55b8-4b08-8512-e8c86b10e317/volumes" Nov 02 14:42:25.075385 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:25.075364 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="537d9533-ce0c-4f93-be2f-684bce472885" path="/var/lib/kubelet/pods/537d9533-ce0c-4f93-be2f-684bce472885/volumes" Nov 02 14:42:35.210652 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:35.210618 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:42:43.788756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:43.788726 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz"] Nov 02 14:42:43.789262 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:43.789084 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" containerID="cri-o://09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae" gracePeriod=30 Nov 02 14:42:44.366510 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:44.366468 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p"] Nov 02 14:42:44.366738 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:44.366691 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" containerID="cri-o://92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446" gracePeriod=30 Nov 02 14:42:49.051210 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.051185 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:42:49.217887 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.217848 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-podinfo\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.217899 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-data\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.217997 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-kube-api-access-6m9zc\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-kube-api-access-6m9zc\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218064 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218035 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-certs\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218120 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-socket\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218163 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-envoy\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218192 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-credential-socket\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218230 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdef57e5-d716-4090-96d0-eaa93c5fc089-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdef57e5-d716-4090-96d0-eaa93c5fc089-istiod-ca-cert\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218234 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-data" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.218410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218288 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-token\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-token\") pod \"fdef57e5-d716-4090-96d0-eaa93c5fc089\" (UID: \"fdef57e5-d716-4090-96d0-eaa93c5fc089\") " Nov 02 14:42:49.218410 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218372 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-certs" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.218642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218469 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-credential-socket" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.218703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218682 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdef57e5-d716-4090-96d0-eaa93c5fc089-istiod-ca-cert" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:49.218762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218735 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.218762 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218757 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.218904 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218772 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.218963 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.218899 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-socket" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.220719 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.220670 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-envoy" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.221160 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.221068 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-podinfo" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:42:49.221766 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.221740 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-token" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:49.222025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.222003 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-kube-api-access-6m9zc" pod "fdef57e5-d716-4090-96d0-eaa93c5fc089" (UID: "fdef57e5-d716-4090-96d0-eaa93c5fc089"). InnerVolumeSpecName "kube-api-access-6m9zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:49.319517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.319483 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9zc\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-kube-api-access-6m9zc\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.319517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.319511 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.319718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.319526 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.319718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.319538 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdef57e5-d716-4090-96d0-eaa93c5fc089-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.319718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.319549 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.319718 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.319559 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdef57e5-d716-4090-96d0-eaa93c5fc089-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.609095 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.609072 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:42:49.679355 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.679324 4431 generic.go:356] "Generic (PLEG): container finished" podID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerID="09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae" exitCode=0 Nov 02 14:42:49.679525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.679378 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" Nov 02 14:42:49.679525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.679403 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" event={"ID":"fdef57e5-d716-4090-96d0-eaa93c5fc089","Type":"ContainerDied","Data":"09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae"} Nov 02 14:42:49.679525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.679460 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz" event={"ID":"fdef57e5-d716-4090-96d0-eaa93c5fc089","Type":"ContainerDied","Data":"4bf5231eca418beee95eb20292df9d7fd5962582a18daeacc0508a08e3d74e97"} Nov 02 14:42:49.679525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.679482 4431 scope.go:122] "RemoveContainer" containerID="09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae" Nov 02 14:42:49.681508 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.681483 4431 generic.go:356] "Generic (PLEG): container finished" podID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerID="92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446" exitCode=0 Nov 02 14:42:49.681645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.681519 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" event={"ID":"ff47094d-1dd4-4433-aaba-5dd88865d905","Type":"ContainerDied","Data":"92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446"} Nov 02 14:42:49.681645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.681533 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" Nov 02 14:42:49.681645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.681560 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p" event={"ID":"ff47094d-1dd4-4433-aaba-5dd88865d905","Type":"ContainerDied","Data":"938859ab3e4414090903387a6d95f43994eea3fb6f8ad6bbff2394901443ff0b"} Nov 02 14:42:49.689616 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.689599 4431 scope.go:122] "RemoveContainer" containerID="09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae" Nov 02 14:42:49.689908 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:49.689883 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae\": container with ID starting with 09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae not found: ID does not exist" containerID="09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae" Nov 02 14:42:49.690011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.689913 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae"} err="failed to get container status \"09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae\": rpc error: code = NotFound desc = could not find container \"09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae\": container with ID starting with 09a107c1df38ec540f0d23304f67221d2d498b092d96a1636c843ac53ce27dae not found: ID does not exist" Nov 02 14:42:49.690011 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.689930 4431 scope.go:122] "RemoveContainer" containerID="92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446" Nov 02 14:42:49.698591 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.698572 4431 scope.go:122] "RemoveContainer" containerID="25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c" Nov 02 14:42:49.706902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.706882 4431 scope.go:122] "RemoveContainer" containerID="92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446" Nov 02 14:42:49.707181 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:49.707160 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446\": container with ID starting with 92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446 not found: ID does not exist" containerID="92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446" Nov 02 14:42:49.707251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.707191 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446"} err="failed to get container status \"92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446\": rpc error: code = NotFound desc = could not find container \"92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446\": container with ID starting with 92acd2419a498ead3e411ebddd85a36a49a8a9fde627e7f2e8d2f632f4f1a446 not found: ID does not exist" Nov 02 14:42:49.707251 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.707214 4431 scope.go:122] "RemoveContainer" containerID="25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c" Nov 02 14:42:49.707455 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:49.707437 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c\": container with ID starting with 25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c not found: ID does not exist" containerID="25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c" Nov 02 14:42:49.707495 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.707462 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c"} err="failed to get container status \"25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c\": rpc error: code = NotFound desc = could not find container \"25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c\": container with ID starting with 25d4372ac57b2ae774653245fd2c7e5b6deb09350fcd8a63d2f4743651972f4c not found: ID does not exist" Nov 02 14:42:49.722857 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.722836 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-certs\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.722958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.722882 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-token\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-token\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.722958 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.722948 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-credential-socket\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.723087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.722986 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-data\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.723087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723034 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-socket\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.723087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723054 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-podinfo\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.723087 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723081 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff47094d-1dd4-4433-aaba-5dd88865d905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff47094d-1dd4-4433-aaba-5dd88865d905-istiod-ca-cert\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.723343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723117 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-certs" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.723343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723122 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-envoy\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.723343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723204 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-kube-api-access-d4qvz\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-kube-api-access-d4qvz\") pod \"ff47094d-1dd4-4433-aaba-5dd88865d905\" (UID: \"ff47094d-1dd4-4433-aaba-5dd88865d905\") " Nov 02 14:42:49.723343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723287 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-data" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.723343 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723316 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-credential-socket" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.723581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723413 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-socket" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.723642 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723589 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff47094d-1dd4-4433-aaba-5dd88865d905-istiod-ca-cert" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:42:49.723704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723669 4431 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.723704 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723689 4431 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.723853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723703 4431 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.723853 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.723719 4431 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.725133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.725101 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-token" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:49.725225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.725152 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-envoy" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:42:49.725225 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.725182 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-podinfo" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 14:42:49.725380 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.725359 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-kube-api-access-d4qvz" pod "ff47094d-1dd4-4433-aaba-5dd88865d905" (UID: "ff47094d-1dd4-4433-aaba-5dd88865d905"). InnerVolumeSpecName "kube-api-access-d4qvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:42:49.736585 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.736540 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz"] Nov 02 14:42:49.738868 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.738848 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-866159-openshift-default-856f7748b9-28fbz"] Nov 02 14:42:49.824886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.824866 4431 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.824886 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.824885 4431 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff47094d-1dd4-4433-aaba-5dd88865d905-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.825055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.824894 4431 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.825055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.824903 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qvz\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-kube-api-access-d4qvz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:49.825055 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:49.824920 4431 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff47094d-1dd4-4433-aaba-5dd88865d905-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:42:50.030889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:50.030862 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p"] Nov 02 14:42:50.035898 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:50.035876 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f72bc6-openshift-default-569b7cb555-gvp4p"] Nov 02 14:42:51.075340 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:51.075297 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" path="/var/lib/kubelet/pods/fdef57e5-d716-4090-96d0-eaa93c5fc089/volumes" Nov 02 14:42:51.075955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:51.075934 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" path="/var/lib/kubelet/pods/ff47094d-1dd4-4433-aaba-5dd88865d905/volumes" Nov 02 14:42:51.282955 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:42:51.282920 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:42:59.012710 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:42:59.012629 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:43:01.052932 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:01.052895 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj"] Nov 02 14:43:01.065989 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:01.065963 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222b8b47-26lvj"] Nov 02 14:43:01.074737 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:01.074708 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03993b8-0205-4fab-ad05-6a86d755dc0d" path="/var/lib/kubelet/pods/e03993b8-0205-4fab-ad05-6a86d755dc0d/volumes" Nov 02 14:43:11.040476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:11.040440 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2"] Nov 02 14:43:11.053537 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:11.053512 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04864e63-8qjl2"] Nov 02 14:43:11.074411 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:11.074372 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fee25df-4efc-4825-892e-5edb2df791f4" path="/var/lib/kubelet/pods/1fee25df-4efc-4825-892e-5edb2df791f4/volumes" Nov 02 14:43:12.830512 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:12.830473 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:43:18.806433 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:18.806400 4431 scope.go:122] "RemoveContainer" containerID="b5837bc788ae9b37255a754b45660809cecd3bda23a128d6aaa1b04000cbbd67" Nov 02 14:43:18.816186 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:18.816159 4431 scope.go:122] "RemoveContainer" containerID="e387ab51dad42fb4a98bac450bdc8461fe40b48319a4d1147a6fa5afa55f5dea" Nov 02 14:43:18.824228 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:18.824212 4431 scope.go:122] "RemoveContainer" containerID="a68f408d8d2fddf9063ba9b211869228a44361e9ebf80477de9e6e81423fb2bd" Nov 02 14:43:18.831679 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:18.831657 4431 scope.go:122] "RemoveContainer" containerID="012abd83c66fa9a94f23947929b4c77b5437c7fee16b248342450e7c21f98b75" Nov 02 14:43:18.839592 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:18.839577 4431 scope.go:122] "RemoveContainer" containerID="47ca70dbc6a7026ba891e2e94d0110fa7738a577a98be9f60c30257871d8b9d1" Nov 02 14:43:18.847571 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:18.847500 4431 scope.go:122] "RemoveContainer" containerID="afee97c6cc078814bf21097634cbaccef4cff019d73d8eff4169c8bb0c25468a" Nov 02 14:43:21.281177 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:43:21.281143 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:43:22.045063 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:22.045029 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67"] Nov 02 14:43:22.048565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:22.048545 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67194783-4ms67"] Nov 02 14:43:23.075013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:23.074985 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfb67e0-8791-4275-8c11-04d8d03d2a32" path="/var/lib/kubelet/pods/abfb67e0-8791-4275-8c11-04d8d03d2a32/volumes" Nov 02 14:43:30.273540 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:30.273509 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mgjcj_1b17862c-afb6-4a79-a21f-54b8e5ea3063/kuadrant-console-plugin/0.log" Nov 02 14:43:30.948827 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:30.948796 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ghltt_4dd71c7a-a298-456d-8d7e-0b1884d987ed/limitador/0.log" Nov 02 14:43:32.468360 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:32.468326 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7xw9r_49b694a0-4451-451b-b77b-f2c1774f478f/manager/0.log" Nov 02 14:43:32.939019 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:32.938991 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fttqq_05fb164b-a69a-4241-97d0-3705e3233692/manager/0.log" Nov 02 14:43:33.572810 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:33.572755 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5db8dc9fd8-76ms5_426ccdb4-cca3-4641-8cc1-af9fffbea7f3/manager/0.log" Nov 02 14:43:33.801803 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:33.801756 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4kw4d_e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550/server/0.log" Nov 02 14:43:34.009185 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:34.009158 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2nb2_bb8c30b8-0631-48c2-8e5e-f7feae79214c/manager/0.log" Nov 02 14:43:41.986412 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:41.986371 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-76ms5"] Nov 02 14:43:41.987146 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:41.986651 4431 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" podUID="426ccdb4-cca3-4641-8cc1-af9fffbea7f3" containerName="manager" containerID="cri-o://e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738" gracePeriod=30 Nov 02 14:43:42.276514 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.276477 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:43:42.375118 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.375068 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-metrics-certs\") pod \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " Nov 02 14:43:42.375307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.375157 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-webhook-cert\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-webhook-cert\") pod \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " Nov 02 14:43:42.375307 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.375256 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-kube-api-access-k2cdt\" (UniqueName: \"kubernetes.io/projected/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-kube-api-access-k2cdt\") pod \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\" (UID: \"426ccdb4-cca3-4641-8cc1-af9fffbea7f3\") " Nov 02 14:43:42.385811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.378137 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-metrics-certs" pod "426ccdb4-cca3-4641-8cc1-af9fffbea7f3" (UID: "426ccdb4-cca3-4641-8cc1-af9fffbea7f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:43:42.385811 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.379119 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-kube-api-access-k2cdt" pod "426ccdb4-cca3-4641-8cc1-af9fffbea7f3" (UID: "426ccdb4-cca3-4641-8cc1-af9fffbea7f3"). InnerVolumeSpecName "kube-api-access-k2cdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:43:42.386935 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.386905 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-webhook-cert" pod "426ccdb4-cca3-4641-8cc1-af9fffbea7f3" (UID: "426ccdb4-cca3-4641-8cc1-af9fffbea7f3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:43:42.476992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.476952 4431 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-metrics-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:43:42.476992 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.476993 4431 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-webhook-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:43:42.477173 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:42.477009 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2cdt\" (UniqueName: \"kubernetes.io/projected/426ccdb4-cca3-4641-8cc1-af9fffbea7f3-kube-api-access-k2cdt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:43:43.014801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.010948 4431 generic.go:356] "Generic (PLEG): container finished" podID="426ccdb4-cca3-4641-8cc1-af9fffbea7f3" containerID="e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738" exitCode=0 Nov 02 14:43:43.014801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.011045 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" event={"ID":"426ccdb4-cca3-4641-8cc1-af9fffbea7f3","Type":"ContainerDied","Data":"e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738"} Nov 02 14:43:43.014801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.011095 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" event={"ID":"426ccdb4-cca3-4641-8cc1-af9fffbea7f3","Type":"ContainerDied","Data":"477655cddfed35295d0cdbf7020f0df68125fd950ffe495e2c89ddaab09f96ba"} Nov 02 14:43:43.014801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.011125 4431 scope.go:122] "RemoveContainer" containerID="e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738" Nov 02 14:43:43.014801 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.011288 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5db8dc9fd8-76ms5" Nov 02 14:43:43.054193 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.054152 4431 scope.go:122] "RemoveContainer" containerID="e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738" Nov 02 14:43:43.054517 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:43:43.054453 4431 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738\": container with ID starting with e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738 not found: ID does not exist" containerID="e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738" Nov 02 14:43:43.054517 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.054492 4431 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738"} err="failed to get container status \"e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738\": rpc error: code = NotFound desc = could not find container \"e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738\": container with ID starting with e0b84b4bbc841fea8974c69375f27c6ecd3062e06aa2c25c7cedcfa78495d738 not found: ID does not exist" Nov 02 14:43:43.130800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.124819 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-76ms5"] Nov 02 14:43:43.142802 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:43.140127 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5db8dc9fd8-76ms5"] Nov 02 14:43:45.081949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:45.081914 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426ccdb4-cca3-4641-8cc1-af9fffbea7f3" path="/var/lib/kubelet/pods/426ccdb4-cca3-4641-8cc1-af9fffbea7f3/volumes" Nov 02 14:43:49.040486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:49.040454 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t"] Nov 02 14:43:49.053665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:49.053637 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32fce375-4gb4t"] Nov 02 14:43:49.055597 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:49.055576 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw"] Nov 02 14:43:49.057283 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:49.057263 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e35f0ffc-g6cfw"] Nov 02 14:43:49.074464 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:49.074436 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6ac095-6aa1-46c9-a337-d8a08cd51fd7" path="/var/lib/kubelet/pods/3d6ac095-6aa1-46c9-a337-d8a08cd51fd7/volumes" Nov 02 14:43:49.074875 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:49.074861 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb998700-a763-44ad-9ce6-ce9d1007ea12" path="/var/lib/kubelet/pods/eb998700-a763-44ad-9ce6-ce9d1007ea12/volumes" Nov 02 14:43:51.282768 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:43:51.282729 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:43:59.032276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:59.032239 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f"] Nov 02 14:43:59.045048 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:59.045016 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4931a8c-n6m7f"] Nov 02 14:43:59.074188 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:43:59.074156 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6f069c-113b-42a2-84d3-c6eb6f4454e0" path="/var/lib/kubelet/pods/1d6f069c-113b-42a2-84d3-c6eb6f4454e0/volumes" Nov 02 14:44:10.036279 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:10.036249 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx"] Nov 02 14:44:10.048660 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:10.048637 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7348e0ec-8svwx"] Nov 02 14:44:11.075137 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:11.075103 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86046477-7a5b-48c0-8a2c-fe3ef1589481" path="/var/lib/kubelet/pods/86046477-7a5b-48c0-8a2c-fe3ef1589481/volumes" Nov 02 14:44:13.031245 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:13.031215 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6"] Nov 02 14:44:13.043265 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:13.043235 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a9b852-x2cp6"] Nov 02 14:44:13.074165 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:13.074134 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ae91ec-a9ab-494c-aea8-b844a8e5a5e4" path="/var/lib/kubelet/pods/67ae91ec-a9ab-494c-aea8-b844a8e5a5e4/volumes" Nov 02 14:44:18.903998 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:18.903962 4431 scope.go:122] "RemoveContainer" containerID="5ba9594be1158df329810d1eb76543d453ae8221a7371842eab14b8a9e687604" Nov 02 14:44:18.912474 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:18.912442 4431 scope.go:122] "RemoveContainer" containerID="ede9fdde13581f64b5719219b383c44e33de2b929b054e15c1ce3bbd7cf676a4" Nov 02 14:44:20.031241 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:20.031210 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v"] Nov 02 14:44:20.044625 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:20.044597 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c605362e-7tc4v"] Nov 02 14:44:21.074701 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:21.074661 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f95370a-57e4-46ef-865b-f3d00c69ab53" path="/var/lib/kubelet/pods/1f95370a-57e4-46ef-865b-f3d00c69ab53/volumes" Nov 02 14:44:21.281273 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:44:21.281238 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:44:35.033021 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:35.032944 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms"] Nov 02 14:44:35.046293 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:35.046267 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c611d23b-mfsms"] Nov 02 14:44:35.075023 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:35.074994 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce1ed78-7686-4253-a226-5aa680fb5b6b" path="/var/lib/kubelet/pods/5ce1ed78-7686-4253-a226-5aa680fb5b6b/volumes" Nov 02 14:44:51.281986 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:44:51.281945 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:44:52.036006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:52.035960 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm"] Nov 02 14:44:52.048833 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:52.048806 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448b04ea-lv8vm"] Nov 02 14:44:53.074270 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:44:53.074238 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b585214-c2fc-4bb8-b1d7-54e12b0667be" path="/var/lib/kubelet/pods/0b585214-c2fc-4bb8-b1d7-54e12b0667be/volumes" Nov 02 14:45:18.941374 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:45:18.941339 4431 scope.go:122] "RemoveContainer" containerID="7519afb0523e1f45015f594c9abe10e86582f4e166844556a7c0529075289ec1" Nov 02 14:45:18.950459 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:45:18.950444 4431 scope.go:122] "RemoveContainer" containerID="f69008dab040a6d65aeea0a6a9f7c9ebcce718cb8b7d05565cac2687da5e35a2" Nov 02 14:45:18.958013 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:45:18.957997 4431 scope.go:122] "RemoveContainer" containerID="e430ec120ac9d00d043e64125cf1752fe1216ee7e5767c9acac93bb21c1ab1d1" Nov 02 14:45:18.965269 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:45:18.965250 4431 scope.go:122] "RemoveContainer" containerID="019d728531959edf8daa215c711f76ddd814e7cb5bd055f631cc0f060dcd6794" Nov 02 14:45:18.972598 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:45:18.972582 4431 scope.go:122] "RemoveContainer" containerID="9c3ae71da29f2db283e695c2af7bcfbb735fc20470e1e2d9a843de8fc131f5e1" Nov 02 14:45:18.980005 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:45:18.979986 4431 scope.go:122] "RemoveContainer" containerID="df2ae00422ffcae9a39a7777f40c3553683c9fc7355faa0bf45931420ffc20ca" Nov 02 14:45:18.987067 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:45:18.987051 4431 scope.go:122] "RemoveContainer" containerID="ee203d2c02d3144dea1daad2c897286215a45de4e054ea9ba4e5440172e8508f" Nov 02 14:45:21.282117 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:45:21.282083 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:45:51.282394 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:45:51.282351 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:46:19.027523 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:46:19.027492 4431 scope.go:122] "RemoveContainer" containerID="86e08d66a51c371307d11e0ad3c57d57c0f59c45f862e01f263c62bc374e172d" Nov 02 14:46:19.035808 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:46:19.035789 4431 scope.go:122] "RemoveContainer" containerID="b610d90c089ce0c94f4accd0eb9e929bfe36c034a535f8e5c117d95afcd7b4d7" Nov 02 14:46:19.523206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:46:19.523175 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:46:21.350463 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:46:21.350435 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:46:42.927848 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:46:42.927814 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:46:48.562395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:46:48.562360 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:46:51.284105 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:46:51.284070 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:47:01.032490 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:01.032455 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4"] Nov 02 14:47:01.046072 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:01.046046 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac4fb65-qpzc4"] Nov 02 14:47:01.074261 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:01.074234 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7867a9ba-77eb-4417-951d-1b1405c4c30c" path="/var/lib/kubelet/pods/7867a9ba-77eb-4417-951d-1b1405c4c30c/volumes" Nov 02 14:47:01.912564 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:01.912524 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:47:07.031553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:07.031519 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p"] Nov 02 14:47:07.044722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:07.044695 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fad2bfd-pj67p"] Nov 02 14:47:07.074483 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:07.074444 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf3f0a9-c255-485a-8aa4-c820eed9630a" path="/var/lib/kubelet/pods/faf3f0a9-c255-485a-8aa4-c820eed9630a/volumes" Nov 02 14:47:13.032000 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:13.031967 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w"] Nov 02 14:47:13.045038 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:13.045011 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dfe8972-kqx9w"] Nov 02 14:47:13.074440 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:13.074406 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977f48d8-3f65-4bb4-82cf-a23f46a3740a" path="/var/lib/kubelet/pods/977f48d8-3f65-4bb4-82cf-a23f46a3740a/volumes" Nov 02 14:47:17.032548 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:17.032513 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp"] Nov 02 14:47:17.045486 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:17.045462 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-566535a3-lrmfp"] Nov 02 14:47:17.074322 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:17.074292 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba931fbf-a2a2-4d1e-bab0-bc555b3280a0" path="/var/lib/kubelet/pods/ba931fbf-a2a2-4d1e-bab0-bc555b3280a0/volumes" Nov 02 14:47:19.301084 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:19.301049 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:47:21.282276 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:47:21.282243 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:47:51.289764 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:47:51.289670 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:47:52.880247 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:52.879868 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:47:52.880247 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:52.879941 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:47:52.945098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:52.942858 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-q8r79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:47:52.945098 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:52.942995 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-q8r79" podUID="bc27444d-3855-4031-8b67-8fe90fb4ae96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:47:52.990949 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:52.990901 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-2m48q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:47:52.991238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:47:52.991192 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-2m48q" podUID="f771e0c6-2f81-4e2f-afd3-20be3bc4eddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:48:19.067563 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:48:19.067528 4431 scope.go:122] "RemoveContainer" containerID="5be065b7647c6b97e30107ce455d2198e1109e7ed9ef6c50a192fab7be2a11c6" Nov 02 14:48:19.076230 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:48:19.076213 4431 scope.go:122] "RemoveContainer" containerID="e4285949c8f125c854e809c84bd9497f8a142fba29a31f559e2583df4c9f866b" Nov 02 14:48:19.083885 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:48:19.083849 4431 scope.go:122] "RemoveContainer" containerID="3c4d8712a8aa7e7134496c99859a8482822d2bdd14674c9967fefb9cbed081f7" Nov 02 14:48:19.094553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:48:19.094532 4431 scope.go:122] "RemoveContainer" containerID="3df2d7e3828e92ad7ce823a4f6f6d714b3ec4b432795af722be78dd37ffc39ce" Nov 02 14:48:21.281537 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:48:21.281491 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:48:51.282678 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:48:51.282638 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:49:21.281944 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:49:21.281911 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:49:23.733933 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:49:23.733901 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:49:38.708525 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:49:38.708488 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:49:51.282492 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:49:51.282445 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:50:03.300666 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:50:03.300629 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:50:10.369163 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:50:10.369129 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:50:21.281860 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:50:21.281821 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:50:23.127796 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:50:23.127744 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:50:28.714889 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:50:28.714806 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:50:34.299800 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:50:34.299622 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:50:39.104594 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:50:39.104562 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:50:51.354874 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:50:51.354837 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:51:15.486581 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:51:15.486544 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:51:21.351723 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:51:21.351659 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:51:23.371988 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:51:23.371957 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:51:51.281964 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:51:51.281927 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:52:00.130446 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:00.130374 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:52:08.160665 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.160631 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7"] Nov 02 14:52:08.161075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161024 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" Nov 02 14:52:08.161075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161036 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" Nov 02 14:52:08.161075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161050 4431 cpu_manager.go:414] "containerMap: removing container" podUID="537d9533-ce0c-4f93-be2f-684bce472885" containerName="main" Nov 02 14:52:08.161075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161055 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="537d9533-ce0c-4f93-be2f-684bce472885" containerName="main" Nov 02 14:52:08.161075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161065 4431 cpu_manager.go:414] "containerMap: removing container" podUID="17b062be-55b8-4b08-8512-e8c86b10e317" containerName="main" Nov 02 14:52:08.161075 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161070 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b062be-55b8-4b08-8512-e8c86b10e317" containerName="main" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161089 4431 cpu_manager.go:414] "containerMap: removing container" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161094 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161101 4431 cpu_manager.go:414] "containerMap: removing container" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161106 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161113 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ba931fbf-a2a2-4d1e-bab0-bc555b3280a0" containerName="revoke-keys" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161118 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba931fbf-a2a2-4d1e-bab0-bc555b3280a0" containerName="revoke-keys" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161134 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161138 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161150 4431 cpu_manager.go:414] "containerMap: removing container" podUID="426ccdb4-cca3-4641-8cc1-af9fffbea7f3" containerName="manager" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161156 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426ccdb4-cca3-4641-8cc1-af9fffbea7f3" containerName="manager" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161213 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d005e58-763a-4c16-98d9-fea871db9498" containerName="main" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161224 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37e3fc7-7ac2-4b12-90a6-78e8730dc69c" containerName="main" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161229 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba931fbf-a2a2-4d1e-bab0-bc555b3280a0" containerName="revoke-keys" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161237 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdef57e5-d716-4090-96d0-eaa93c5fc089" containerName="istio-proxy" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161243 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="426ccdb4-cca3-4641-8cc1-af9fffbea7f3" containerName="manager" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161250 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161258 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="537d9533-ce0c-4f93-be2f-684bce472885" containerName="main" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161264 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" Nov 02 14:52:08.161302 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.161270 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b062be-55b8-4b08-8512-e8c86b10e317" containerName="main" Nov 02 14:52:08.164499 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.164483 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.180951 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.180932 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:52:08.194025 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.194000 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 14:52:08.195756 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.195735 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7"] Nov 02 14:52:08.255006 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.254981 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.255134 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.255022 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5zt\" (UniqueName: \"kubernetes.io/projected/1ac2ad5e-61a2-434e-820f-e4f005369da0-kube-api-access-5v5zt\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.255207 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.255140 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.355793 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.355757 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zt\" (UniqueName: \"kubernetes.io/projected/1ac2ad5e-61a2-434e-820f-e4f005369da0-kube-api-access-5v5zt\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.355919 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.355865 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.355994 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.355956 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.356528 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.356509 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.358195 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.358177 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.380298 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.380275 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5zt\" (UniqueName: \"kubernetes.io/projected/1ac2ad5e-61a2-434e-820f-e4f005369da0-kube-api-access-5v5zt\") pod \"maas-api-revoke-sub-e29ea64c-zwxl7\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.476560 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.476497 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:08.773214 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:08.773186 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7"] Nov 02 14:52:08.776021 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:52:08.775991 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac2ad5e_61a2_434e_820f_e4f005369da0.slice/crio-4b403cca2c28e4a47d13b0bf1741f594d0a0051d006c97ed7f56a68a2864415b WatchSource:0}: Error finding container 4b403cca2c28e4a47d13b0bf1741f594d0a0051d006c97ed7f56a68a2864415b: Status 404 returned error can't find the container with id 4b403cca2c28e4a47d13b0bf1741f594d0a0051d006c97ed7f56a68a2864415b Nov 02 14:52:09.684128 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:09.684103 4431 generic.go:356] "Generic (PLEG): container finished" podID="1ac2ad5e-61a2-434e-820f-e4f005369da0" containerID="d94ce26f79016c4cc72a5d7e536c803bb8fd69e34cf53018128249f9ea3b9e7e" exitCode=0 Nov 02 14:52:09.684460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:09.684138 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" event={"ID":"1ac2ad5e-61a2-434e-820f-e4f005369da0","Type":"ContainerDied","Data":"d94ce26f79016c4cc72a5d7e536c803bb8fd69e34cf53018128249f9ea3b9e7e"} Nov 02 14:52:09.684460 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:09.684177 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" event={"ID":"1ac2ad5e-61a2-434e-820f-e4f005369da0","Type":"ContainerStarted","Data":"4b403cca2c28e4a47d13b0bf1741f594d0a0051d006c97ed7f56a68a2864415b"} Nov 02 14:52:10.820829 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:10.820802 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:10.979255 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:10.979196 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-openshift-ca\") pod \"1ac2ad5e-61a2-434e-820f-e4f005369da0\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " Nov 02 14:52:10.979394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:10.979287 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac2ad5e-61a2-434e-820f-e4f005369da0-kube-api-access-5v5zt\" (UniqueName: \"kubernetes.io/projected/1ac2ad5e-61a2-434e-820f-e4f005369da0-kube-api-access-5v5zt\") pod \"1ac2ad5e-61a2-434e-820f-e4f005369da0\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " Nov 02 14:52:10.979394 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:10.979316 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-xks-ca\") pod \"1ac2ad5e-61a2-434e-820f-e4f005369da0\" (UID: \"1ac2ad5e-61a2-434e-820f-e4f005369da0\") " Nov 02 14:52:10.979539 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:10.979518 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-openshift-ca" pod "1ac2ad5e-61a2-434e-820f-e4f005369da0" (UID: "1ac2ad5e-61a2-434e-820f-e4f005369da0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:52:10.979691 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:10.979668 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:52:10.981346 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:10.981326 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-xks-ca" pod "1ac2ad5e-61a2-434e-820f-e4f005369da0" (UID: "1ac2ad5e-61a2-434e-820f-e4f005369da0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:52:10.981482 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:10.981458 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac2ad5e-61a2-434e-820f-e4f005369da0-kube-api-access-5v5zt" pod "1ac2ad5e-61a2-434e-820f-e4f005369da0" (UID: "1ac2ad5e-61a2-434e-820f-e4f005369da0"). InnerVolumeSpecName "kube-api-access-5v5zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:52:11.080884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:11.080856 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5zt\" (UniqueName: \"kubernetes.io/projected/1ac2ad5e-61a2-434e-820f-e4f005369da0-kube-api-access-5v5zt\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:52:11.080884 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:11.080881 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac2ad5e-61a2-434e-820f-e4f005369da0-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:52:11.694577 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:11.694549 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" Nov 02 14:52:11.694799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:11.694586 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7" event={"ID":"1ac2ad5e-61a2-434e-820f-e4f005369da0","Type":"ContainerDied","Data":"4b403cca2c28e4a47d13b0bf1741f594d0a0051d006c97ed7f56a68a2864415b"} Nov 02 14:52:11.694799 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:11.694636 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b403cca2c28e4a47d13b0bf1741f594d0a0051d006c97ed7f56a68a2864415b" Nov 02 14:52:19.603722 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:19.603684 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:52:21.281203 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:52:21.281168 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:52:24.685324 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.685292 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf"] Nov 02 14:52:24.685836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.685750 4431 cpu_manager.go:414] "containerMap: removing container" podUID="1ac2ad5e-61a2-434e-820f-e4f005369da0" containerName="revoke-keys" Nov 02 14:52:24.685836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.685767 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac2ad5e-61a2-434e-820f-e4f005369da0" containerName="revoke-keys" Nov 02 14:52:24.685836 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.685833 4431 cpu_manager.go:414] "containerMap: removing container" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" Nov 02 14:52:24.685955 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.685842 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff47094d-1dd4-4433-aaba-5dd88865d905" containerName="istio-proxy" Nov 02 14:52:24.685990 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.685964 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac2ad5e-61a2-434e-820f-e4f005369da0" containerName="revoke-keys" Nov 02 14:52:24.688703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.688688 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.720312 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.720279 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:52:24.720476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.720402 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 14:52:24.722457 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.722220 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf"] Nov 02 14:52:24.738530 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.738501 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbjb7\" (UniqueName: \"kubernetes.io/projected/16567b68-ebfe-4e22-b928-1b539c708fb0-kube-api-access-wbjb7\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.738657 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.738539 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.738657 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.738628 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.839395 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.839369 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbjb7\" (UniqueName: \"kubernetes.io/projected/16567b68-ebfe-4e22-b928-1b539c708fb0-kube-api-access-wbjb7\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.839532 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.839426 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.839532 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.839484 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.840149 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.840130 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.841946 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.841925 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:24.864264 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:24.864238 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbjb7\" (UniqueName: \"kubernetes.io/projected/16567b68-ebfe-4e22-b928-1b539c708fb0-kube-api-access-wbjb7\") pod \"maas-api-revoke-sub-93d6b4dd-csrkf\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:25.001017 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:25.000966 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:25.292602 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:25.292499 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:52:25.295550 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:25.295433 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf"] Nov 02 14:52:25.298444 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:52:25.298420 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16567b68_ebfe_4e22_b928_1b539c708fb0.slice/crio-b722f3841e57ea7130c69dd9331ef3d988738f8cb29f8cedff1e4273d44580fa WatchSource:0}: Error finding container b722f3841e57ea7130c69dd9331ef3d988738f8cb29f8cedff1e4273d44580fa: Status 404 returned error can't find the container with id b722f3841e57ea7130c69dd9331ef3d988738f8cb29f8cedff1e4273d44580fa Nov 02 14:52:25.767138 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:25.767099 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" event={"ID":"16567b68-ebfe-4e22-b928-1b539c708fb0","Type":"ContainerStarted","Data":"e6a7bf3f80a73c4c7a1f433392261e5ad69f3c2287eeb60f79485dde9038c8f4"} Nov 02 14:52:25.767645 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:25.767166 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" event={"ID":"16567b68-ebfe-4e22-b928-1b539c708fb0","Type":"ContainerStarted","Data":"b722f3841e57ea7130c69dd9331ef3d988738f8cb29f8cedff1e4273d44580fa"} Nov 02 14:52:25.812869 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:25.812815 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" podStartSLOduration=1.812795728 podStartE2EDuration="1.812795728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:52:25.810000841 +0000 UTC m=+3310.847105774" watchObservedRunningTime="2026-11-02 14:52:25.812795728 +0000 UTC m=+3310.849900660" Nov 02 14:52:26.785805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:26.785400 4431 generic.go:356] "Generic (PLEG): container finished" podID="16567b68-ebfe-4e22-b928-1b539c708fb0" containerID="e6a7bf3f80a73c4c7a1f433392261e5ad69f3c2287eeb60f79485dde9038c8f4" exitCode=0 Nov 02 14:52:26.785805 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:26.785518 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" event={"ID":"16567b68-ebfe-4e22-b928-1b539c708fb0","Type":"ContainerDied","Data":"e6a7bf3f80a73c4c7a1f433392261e5ad69f3c2287eeb60f79485dde9038c8f4"} Nov 02 14:52:28.140968 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.140942 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:28.169417 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.169390 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-xks-ca\") pod \"16567b68-ebfe-4e22-b928-1b539c708fb0\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " Nov 02 14:52:28.169553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.169533 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16567b68-ebfe-4e22-b928-1b539c708fb0-kube-api-access-wbjb7\" (UniqueName: \"kubernetes.io/projected/16567b68-ebfe-4e22-b928-1b539c708fb0-kube-api-access-wbjb7\") pod \"16567b68-ebfe-4e22-b928-1b539c708fb0\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " Nov 02 14:52:28.169671 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.169654 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-openshift-ca\") pod \"16567b68-ebfe-4e22-b928-1b539c708fb0\" (UID: \"16567b68-ebfe-4e22-b928-1b539c708fb0\") " Nov 02 14:52:28.170069 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.170027 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-openshift-ca" pod "16567b68-ebfe-4e22-b928-1b539c708fb0" (UID: "16567b68-ebfe-4e22-b928-1b539c708fb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:52:28.171575 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.171540 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-xks-ca" pod "16567b68-ebfe-4e22-b928-1b539c708fb0" (UID: "16567b68-ebfe-4e22-b928-1b539c708fb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:52:28.172278 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.172254 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16567b68-ebfe-4e22-b928-1b539c708fb0-kube-api-access-wbjb7" pod "16567b68-ebfe-4e22-b928-1b539c708fb0" (UID: "16567b68-ebfe-4e22-b928-1b539c708fb0"). InnerVolumeSpecName "kube-api-access-wbjb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:52:28.271289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.271262 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbjb7\" (UniqueName: \"kubernetes.io/projected/16567b68-ebfe-4e22-b928-1b539c708fb0-kube-api-access-wbjb7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:52:28.271289 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.271285 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:52:28.271475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.271296 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16567b68-ebfe-4e22-b928-1b539c708fb0-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:52:28.802871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.802826 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" event={"ID":"16567b68-ebfe-4e22-b928-1b539c708fb0","Type":"ContainerDied","Data":"b722f3841e57ea7130c69dd9331ef3d988738f8cb29f8cedff1e4273d44580fa"} Nov 02 14:52:28.802871 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.802851 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf" Nov 02 14:52:28.803106 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:28.802884 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b722f3841e57ea7130c69dd9331ef3d988738f8cb29f8cedff1e4273d44580fa" Nov 02 14:52:42.686578 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:42.686544 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:52:51.281795 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:52:51.281743 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:52:59.611862 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:52:59.611832 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:53:21.282321 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:53:21.282278 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:53:25.321461 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:53:25.321425 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:53:39.723524 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:53:39.723440 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:53:47.560402 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:53:47.560367 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:53:51.281094 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:53:51.281058 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:53:55.930770 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:53:55.930736 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:54:00.683729 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:54:00.683694 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:54:16.331248 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:54:16.331214 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:54:21.281882 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:54:21.281837 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:54:26.148553 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:54:26.148520 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:54:51.281211 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:54:51.281174 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:55:09.617579 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:55:09.617536 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:55:21.281330 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:55:21.281295 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:55:51.281947 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:55:51.281912 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:55:55.211849 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:55:55.211818 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:56:21.282334 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:56:21.282295 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:56:25.415568 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:56:25.415536 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:56:51.282046 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:56:51.282007 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:57:11.031655 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:11.031617 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7"] Nov 02 14:57:11.044745 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:11.044719 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ea64c-zwxl7"] Nov 02 14:57:11.074565 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:11.074526 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac2ad5e-61a2-434e-820f-e4f005369da0" path="/var/lib/kubelet/pods/1ac2ad5e-61a2-434e-820f-e4f005369da0/volumes" Nov 02 14:57:21.283013 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:57:21.282973 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:57:28.030902 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:28.030867 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf"] Nov 02 14:57:28.043703 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:28.043679 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d6b4dd-csrkf"] Nov 02 14:57:29.073876 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:29.073836 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16567b68-ebfe-4e22-b928-1b539c708fb0" path="/var/lib/kubelet/pods/16567b68-ebfe-4e22-b928-1b539c708fb0/volumes" Nov 02 14:57:30.391016 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.390986 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds"] Nov 02 14:57:30.391398 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.391392 4431 cpu_manager.go:414] "containerMap: removing container" podUID="16567b68-ebfe-4e22-b928-1b539c708fb0" containerName="revoke-keys" Nov 02 14:57:30.391441 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.391402 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16567b68-ebfe-4e22-b928-1b539c708fb0" containerName="revoke-keys" Nov 02 14:57:30.391478 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.391462 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="16567b68-ebfe-4e22-b928-1b539c708fb0" containerName="revoke-keys" Nov 02 14:57:30.394114 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.394096 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.414435 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.414415 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds"] Nov 02 14:57:30.425834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.425815 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 14:57:30.425834 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.425827 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 14:57:30.538360 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.538332 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.538470 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.538369 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25pcx\" (UniqueName: \"kubernetes.io/projected/fd3a56c4-8d79-494b-962a-9a18dfc91420-kube-api-access-25pcx\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.538518 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.538471 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.639476 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.639445 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.639588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.639486 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25pcx\" (UniqueName: \"kubernetes.io/projected/fd3a56c4-8d79-494b-962a-9a18dfc91420-kube-api-access-25pcx\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.639588 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.639550 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.640238 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.640215 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.641953 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.641907 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.664205 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.664175 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25pcx\" (UniqueName: \"kubernetes.io/projected/fd3a56c4-8d79-494b-962a-9a18dfc91420-kube-api-access-25pcx\") pod \"maas-api-revoke-sub-88af6301-hl7ds\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:30.706841 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:30.706819 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:31.002076 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:31.002053 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds"] Nov 02 14:57:31.004334 ip-10-0-134-9 kubenswrapper[4431]: W1102 14:57:31.004306 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3a56c4_8d79_494b_962a_9a18dfc91420.slice/crio-effdcecbcc258c4e8dc21eb98901f75cca2a5efc5fc1c9c48fb437a3e96f70f6 WatchSource:0}: Error finding container effdcecbcc258c4e8dc21eb98901f75cca2a5efc5fc1c9c48fb437a3e96f70f6: Status 404 returned error can't find the container with id effdcecbcc258c4e8dc21eb98901f75cca2a5efc5fc1c9c48fb437a3e96f70f6 Nov 02 14:57:31.347220 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:31.347182 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" event={"ID":"fd3a56c4-8d79-494b-962a-9a18dfc91420","Type":"ContainerStarted","Data":"78ac4f138645715494d25a99d75db8daf7342a68466a1e7b9c41f0f06f6b6e83"} Nov 02 14:57:31.347377 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:31.347246 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" event={"ID":"fd3a56c4-8d79-494b-962a-9a18dfc91420","Type":"ContainerStarted","Data":"effdcecbcc258c4e8dc21eb98901f75cca2a5efc5fc1c9c48fb437a3e96f70f6"} Nov 02 14:57:31.397748 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:31.397694 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" podStartSLOduration=1.397680236 podStartE2EDuration="1.397680236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:31.395341074 +0000 UTC m=+3616.432446006" watchObservedRunningTime="2026-11-02 14:57:31.397680236 +0000 UTC m=+3616.434785165" Nov 02 14:57:32.097965 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:32.097896 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:57:32.356356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:32.356211 4431 generic.go:356] "Generic (PLEG): container finished" podID="fd3a56c4-8d79-494b-962a-9a18dfc91420" containerID="78ac4f138645715494d25a99d75db8daf7342a68466a1e7b9c41f0f06f6b6e83" exitCode=0 Nov 02 14:57:32.356356 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:32.356299 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" event={"ID":"fd3a56c4-8d79-494b-962a-9a18dfc91420","Type":"ContainerDied","Data":"78ac4f138645715494d25a99d75db8daf7342a68466a1e7b9c41f0f06f6b6e83"} Nov 02 14:57:33.494177 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.494154 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:33.663970 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.663940 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-xks-ca\") pod \"fd3a56c4-8d79-494b-962a-9a18dfc91420\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " Nov 02 14:57:33.664133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.664002 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3a56c4-8d79-494b-962a-9a18dfc91420-kube-api-access-25pcx\" (UniqueName: \"kubernetes.io/projected/fd3a56c4-8d79-494b-962a-9a18dfc91420-kube-api-access-25pcx\") pod \"fd3a56c4-8d79-494b-962a-9a18dfc91420\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " Nov 02 14:57:33.664133 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.664038 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-openshift-ca\") pod \"fd3a56c4-8d79-494b-962a-9a18dfc91420\" (UID: \"fd3a56c4-8d79-494b-962a-9a18dfc91420\") " Nov 02 14:57:33.664437 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.664416 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-openshift-ca" pod "fd3a56c4-8d79-494b-962a-9a18dfc91420" (UID: "fd3a56c4-8d79-494b-962a-9a18dfc91420"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:57:33.666206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.666178 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-xks-ca" pod "fd3a56c4-8d79-494b-962a-9a18dfc91420" (UID: "fd3a56c4-8d79-494b-962a-9a18dfc91420"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:57:33.666206 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.666197 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3a56c4-8d79-494b-962a-9a18dfc91420-kube-api-access-25pcx" pod "fd3a56c4-8d79-494b-962a-9a18dfc91420" (UID: "fd3a56c4-8d79-494b-962a-9a18dfc91420"). InnerVolumeSpecName "kube-api-access-25pcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:57:33.765475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.765450 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:57:33.765475 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.765472 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25pcx\" (UniqueName: \"kubernetes.io/projected/fd3a56c4-8d79-494b-962a-9a18dfc91420-kube-api-access-25pcx\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:57:33.765628 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:33.765481 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3a56c4-8d79-494b-962a-9a18dfc91420-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 14:57:34.366584 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:34.366555 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" Nov 02 14:57:34.366751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:34.366593 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds" event={"ID":"fd3a56c4-8d79-494b-962a-9a18dfc91420","Type":"ContainerDied","Data":"effdcecbcc258c4e8dc21eb98901f75cca2a5efc5fc1c9c48fb437a3e96f70f6"} Nov 02 14:57:34.366751 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:34.366639 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="effdcecbcc258c4e8dc21eb98901f75cca2a5efc5fc1c9c48fb437a3e96f70f6" Nov 02 14:57:46.121973 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:57:46.121939 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:57:51.282438 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:57:51.282398 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:58:10.028837 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:58:10.028803 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:58:19.184276 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:58:19.184245 4431 scope.go:122] "RemoveContainer" containerID="d94ce26f79016c4cc72a5d7e536c803bb8fd69e34cf53018128249f9ea3b9e7e" Nov 02 14:58:21.282353 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:58:21.282314 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:58:51.281619 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:58:51.281584 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:58:59.734867 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:58:59.734830 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 14:59:19.207285 ip-10-0-134-9 kubenswrapper[4431]: I1102 14:59:19.207254 4431 scope.go:122] "RemoveContainer" containerID="e6a7bf3f80a73c4c7a1f433392261e5ad69f3c2287eeb60f79485dde9038c8f4" Nov 02 14:59:21.282663 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:59:21.282628 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 14:59:51.283804 ip-10-0-134-9 kubenswrapper[4431]: E1102 14:59:51.283699 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:00:21.282220 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:00:21.282173 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:00:47.697247 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:00:47.697208 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:00:51.282208 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:00:51.282160 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:00:54.536009 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:00:54.535954 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:01:10.295685 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:01:10.295611 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:01:21.280809 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:01:21.280759 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:01:26.486044 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:01:26.486006 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:01:49.993047 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:01:49.993010 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:01:51.350305 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:01:51.350231 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:02:00.317815 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:00.317768 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:02:05.104028 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:05.103995 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:02:07.878899 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:07.878868 4431 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55fbd65575-t9gjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:02:07.879479 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:07.878963 4431 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55fbd65575-t9gjp" podUID="fefae9cc-8fdb-4c2a-9029-5218b3f2ea97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:02:21.281227 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:02:21.281190 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:02:22.118851 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:22.118819 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:02:27.607333 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:27.607302 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:02:32.211093 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:32.211058 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:02:34.052302 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:34.052268 4431 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds"] Nov 02 15:02:34.073466 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:34.073421 4431 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88af6301-hl7ds"] Nov 02 15:02:35.074699 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:35.074664 4431 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3a56c4-8d79-494b-962a-9a18dfc91420" path="/var/lib/kubelet/pods/fd3a56c4-8d79-494b-962a-9a18dfc91420/volumes" Nov 02 15:02:44.219633 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:44.219597 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:02:49.824971 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:49.824934 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:02:51.282826 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:02:51.282763 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:02:57.832387 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:02:57.832344 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:03:03.507737 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:03:03.507704 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:03:21.281409 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:03:21.281374 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:03:21.587070 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:03:21.586976 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:03:27.857923 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:03:27.857884 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:03:51.281220 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:03:51.281183 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:04:00.909215 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:04:00.909182 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:04:19.207337 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:04:19.207301 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:04:19.252940 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:04:19.252914 4431 scope.go:122] "RemoveContainer" containerID="78ac4f138645715494d25a99d75db8daf7342a68466a1e7b9c41f0f06f6b6e83" Nov 02 15:04:21.349384 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:04:21.349346 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:04:29.425720 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:04:29.425687 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:04:51.288169 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:04:51.287822 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:04:59.106795 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:04:59.106741 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:05:13.817631 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:05:13.817592 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:05:21.287382 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:05:21.287345 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:05:29.237140 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:05:29.237059 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:05:51.282766 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:05:51.282727 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:05:55.582004 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:05:55.581975 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:06:06.824530 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:06:06.824496 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:06:21.281901 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:06:21.281862 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:06:39.993036 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:06:39.993005 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:06:51.281763 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:06:51.281728 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:07:00.918432 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:00.918348 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:07:17.394612 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:17.394580 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:07:21.280920 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:07:21.280887 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:07:32.404601 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:32.404560 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:07:44.829016 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:44.828976 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:07:51.280743 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:07:51.280707 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:07:53.331058 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.331025 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw"] Nov 02 15:07:53.331505 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.331486 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fd3a56c4-8d79-494b-962a-9a18dfc91420" containerName="revoke-keys" Nov 02 15:07:53.331556 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.331508 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3a56c4-8d79-494b-962a-9a18dfc91420" containerName="revoke-keys" Nov 02 15:07:53.331607 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.331598 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3a56c4-8d79-494b-962a-9a18dfc91420" containerName="revoke-keys" Nov 02 15:07:53.334674 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.334659 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.354959 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.354939 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:07:53.354959 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.354944 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 15:07:53.370107 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.370083 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw"] Nov 02 15:07:53.442474 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.442445 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.442629 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.442483 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.442629 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.442567 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtb4n\" (UniqueName: \"kubernetes.io/projected/f5a3d55c-a10c-487f-8d73-0b6c8a370967-kube-api-access-dtb4n\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.543480 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.543451 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.543616 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.543492 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.543616 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.543528 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtb4n\" (UniqueName: \"kubernetes.io/projected/f5a3d55c-a10c-487f-8d73-0b6c8a370967-kube-api-access-dtb4n\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.544188 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.544164 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.545789 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.545754 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.568468 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.568445 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtb4n\" (UniqueName: \"kubernetes.io/projected/f5a3d55c-a10c-487f-8d73-0b6c8a370967-kube-api-access-dtb4n\") pod \"maas-api-revoke-sub-78cad57d-c7ktw\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.647076 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.647048 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:53.945685 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:53.945656 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw"] Nov 02 15:07:53.948358 ip-10-0-134-9 kubenswrapper[4431]: W1102 15:07:53.948329 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a3d55c_a10c_487f_8d73_0b6c8a370967.slice/crio-718e579c5cbc17f4f7acce139e5094a2a5f7f875cf2aaf08e99cb576d56518e0 WatchSource:0}: Error finding container 718e579c5cbc17f4f7acce139e5094a2a5f7f875cf2aaf08e99cb576d56518e0: Status 404 returned error can't find the container with id 718e579c5cbc17f4f7acce139e5094a2a5f7f875cf2aaf08e99cb576d56518e0 Nov 02 15:07:54.359988 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:54.359946 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" event={"ID":"f5a3d55c-a10c-487f-8d73-0b6c8a370967","Type":"ContainerStarted","Data":"77784c66f586ea77d8233c6f69b5c9d8699f9c9d2bd429905a5a4ecf254475fc"} Nov 02 15:07:54.360350 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:54.360009 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" event={"ID":"f5a3d55c-a10c-487f-8d73-0b6c8a370967","Type":"ContainerStarted","Data":"718e579c5cbc17f4f7acce139e5094a2a5f7f875cf2aaf08e99cb576d56518e0"} Nov 02 15:07:54.403743 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:54.403699 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" podStartSLOduration=1.403687391 podStartE2EDuration="1.403687391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:54.403064487 +0000 UTC m=+4239.440169444" watchObservedRunningTime="2026-11-02 15:07:54.403687391 +0000 UTC m=+4239.440792322" Nov 02 15:07:55.365626 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:55.365590 4431 generic.go:356] "Generic (PLEG): container finished" podID="f5a3d55c-a10c-487f-8d73-0b6c8a370967" containerID="77784c66f586ea77d8233c6f69b5c9d8699f9c9d2bd429905a5a4ecf254475fc" exitCode=0 Nov 02 15:07:55.366032 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:55.365674 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" event={"ID":"f5a3d55c-a10c-487f-8d73-0b6c8a370967","Type":"ContainerDied","Data":"77784c66f586ea77d8233c6f69b5c9d8699f9c9d2bd429905a5a4ecf254475fc"} Nov 02 15:07:56.505011 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.504988 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:56.569996 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.569964 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-openshift-ca\") pod \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " Nov 02 15:07:56.570132 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.570072 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a3d55c-a10c-487f-8d73-0b6c8a370967-kube-api-access-dtb4n\" (UniqueName: \"kubernetes.io/projected/f5a3d55c-a10c-487f-8d73-0b6c8a370967-kube-api-access-dtb4n\") pod \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " Nov 02 15:07:56.570181 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.570147 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-xks-ca\") pod \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\" (UID: \"f5a3d55c-a10c-487f-8d73-0b6c8a370967\") " Nov 02 15:07:56.570295 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.570274 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-openshift-ca" pod "f5a3d55c-a10c-487f-8d73-0b6c8a370967" (UID: "f5a3d55c-a10c-487f-8d73-0b6c8a370967"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:07:56.570499 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.570478 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:07:56.572176 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.572147 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-xks-ca" pod "f5a3d55c-a10c-487f-8d73-0b6c8a370967" (UID: "f5a3d55c-a10c-487f-8d73-0b6c8a370967"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:07:56.572276 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.572253 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a3d55c-a10c-487f-8d73-0b6c8a370967-kube-api-access-dtb4n" pod "f5a3d55c-a10c-487f-8d73-0b6c8a370967" (UID: "f5a3d55c-a10c-487f-8d73-0b6c8a370967"). InnerVolumeSpecName "kube-api-access-dtb4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:56.670994 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.670962 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtb4n\" (UniqueName: \"kubernetes.io/projected/f5a3d55c-a10c-487f-8d73-0b6c8a370967-kube-api-access-dtb4n\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:07:56.670994 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:56.670994 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a3d55c-a10c-487f-8d73-0b6c8a370967-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:07:57.375173 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:57.375100 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" Nov 02 15:07:57.375340 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:57.375100 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cad57d-c7ktw" event={"ID":"f5a3d55c-a10c-487f-8d73-0b6c8a370967","Type":"ContainerDied","Data":"718e579c5cbc17f4f7acce139e5094a2a5f7f875cf2aaf08e99cb576d56518e0"} Nov 02 15:07:57.375340 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:07:57.375212 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718e579c5cbc17f4f7acce139e5094a2a5f7f875cf2aaf08e99cb576d56518e0" Nov 02 15:08:03.549969 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.549935 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks"] Nov 02 15:08:03.550391 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.550327 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f5a3d55c-a10c-487f-8d73-0b6c8a370967" containerName="revoke-keys" Nov 02 15:08:03.550391 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.550342 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a3d55c-a10c-487f-8d73-0b6c8a370967" containerName="revoke-keys" Nov 02 15:08:03.550486 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.550452 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a3d55c-a10c-487f-8d73-0b6c8a370967" containerName="revoke-keys" Nov 02 15:08:03.554021 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.553994 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.575374 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.575351 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:08:03.588015 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.587986 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 15:08:03.591055 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.591029 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks"] Nov 02 15:08:03.632647 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.632614 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.632841 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.632667 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.632841 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.632768 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589ft\" (UniqueName: \"kubernetes.io/projected/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-kube-api-access-589ft\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.733898 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.733865 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589ft\" (UniqueName: \"kubernetes.io/projected/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-kube-api-access-589ft\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.734095 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.733942 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.734095 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.733983 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.734605 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.734586 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.736408 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.736384 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.759418 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.759393 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-589ft\" (UniqueName: \"kubernetes.io/projected/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-kube-api-access-589ft\") pod \"maas-api-revoke-sub-277d08f7-s6wks\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:03.869972 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:03.869881 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:04.167182 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:04.167160 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks"] Nov 02 15:08:04.169624 ip-10-0-134-9 kubenswrapper[4431]: W1102 15:08:04.169596 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496ef5f8_7a43_4f6b_8e0a_366aa83be69c.slice/crio-61ff07babc5e03681d8f69ea136d010f5c890981f5ec887f7968dd9e5af50e8e WatchSource:0}: Error finding container 61ff07babc5e03681d8f69ea136d010f5c890981f5ec887f7968dd9e5af50e8e: Status 404 returned error can't find the container with id 61ff07babc5e03681d8f69ea136d010f5c890981f5ec887f7968dd9e5af50e8e Nov 02 15:08:04.407551 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:04.407523 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" event={"ID":"496ef5f8-7a43-4f6b-8e0a-366aa83be69c","Type":"ContainerStarted","Data":"d83a1491d59d4a2132715d647c3140423e6bfa198c6b09f0a8bbbfcdab544566"} Nov 02 15:08:04.407670 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:04.407583 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" event={"ID":"496ef5f8-7a43-4f6b-8e0a-366aa83be69c","Type":"ContainerStarted","Data":"61ff07babc5e03681d8f69ea136d010f5c890981f5ec887f7968dd9e5af50e8e"} Nov 02 15:08:04.451522 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:04.451452 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" podStartSLOduration=1.451440165 podStartE2EDuration="1.451440165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:04.450123562 +0000 UTC m=+4249.487228497" watchObservedRunningTime="2026-11-02 15:08:04.451440165 +0000 UTC m=+4249.488545097" Nov 02 15:08:05.414451 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:05.414422 4431 generic.go:356] "Generic (PLEG): container finished" podID="496ef5f8-7a43-4f6b-8e0a-366aa83be69c" containerID="d83a1491d59d4a2132715d647c3140423e6bfa198c6b09f0a8bbbfcdab544566" exitCode=0 Nov 02 15:08:05.414854 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:05.414499 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" event={"ID":"496ef5f8-7a43-4f6b-8e0a-366aa83be69c","Type":"ContainerDied","Data":"d83a1491d59d4a2132715d647c3140423e6bfa198c6b09f0a8bbbfcdab544566"} Nov 02 15:08:06.553392 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.553368 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:06.661142 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.661110 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-openshift-ca\") pod \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " Nov 02 15:08:06.661333 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.661183 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-kube-api-access-589ft\" (UniqueName: \"kubernetes.io/projected/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-kube-api-access-589ft\") pod \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " Nov 02 15:08:06.661333 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.661288 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-xks-ca\") pod \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\" (UID: \"496ef5f8-7a43-4f6b-8e0a-366aa83be69c\") " Nov 02 15:08:06.661479 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.661459 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-openshift-ca" pod "496ef5f8-7a43-4f6b-8e0a-366aa83be69c" (UID: "496ef5f8-7a43-4f6b-8e0a-366aa83be69c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:08:06.661673 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.661660 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:08:06.663167 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.663144 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-xks-ca" pod "496ef5f8-7a43-4f6b-8e0a-366aa83be69c" (UID: "496ef5f8-7a43-4f6b-8e0a-366aa83be69c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:08:06.663279 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.663265 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-kube-api-access-589ft" pod "496ef5f8-7a43-4f6b-8e0a-366aa83be69c" (UID: "496ef5f8-7a43-4f6b-8e0a-366aa83be69c"). InnerVolumeSpecName "kube-api-access-589ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:08:06.763121 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.763093 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589ft\" (UniqueName: \"kubernetes.io/projected/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-kube-api-access-589ft\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:08:06.763121 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:06.763119 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496ef5f8-7a43-4f6b-8e0a-366aa83be69c-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:08:07.424225 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:07.424187 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" Nov 02 15:08:07.424225 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:07.424201 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277d08f7-s6wks" event={"ID":"496ef5f8-7a43-4f6b-8e0a-366aa83be69c","Type":"ContainerDied","Data":"61ff07babc5e03681d8f69ea136d010f5c890981f5ec887f7968dd9e5af50e8e"} Nov 02 15:08:07.424451 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:07.424276 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ff07babc5e03681d8f69ea136d010f5c890981f5ec887f7968dd9e5af50e8e" Nov 02 15:08:14.832233 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:14.832197 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:08:21.283627 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:08:21.283592 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:08:29.704560 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:29.704478 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:08:35.632170 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.632130 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps"] Nov 02 15:08:35.632721 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.632700 4431 cpu_manager.go:414] "containerMap: removing container" podUID="496ef5f8-7a43-4f6b-8e0a-366aa83be69c" containerName="revoke-keys" Nov 02 15:08:35.632837 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.632722 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496ef5f8-7a43-4f6b-8e0a-366aa83be69c" containerName="revoke-keys" Nov 02 15:08:35.632909 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.632894 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="496ef5f8-7a43-4f6b-8e0a-366aa83be69c" containerName="revoke-keys" Nov 02 15:08:35.636343 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.636319 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.656291 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.656223 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps"] Nov 02 15:08:35.666821 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.666797 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:08:35.666958 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.666936 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 15:08:35.757659 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.757632 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfs9b\" (UniqueName: \"kubernetes.io/projected/770f4175-e525-4a6f-b54d-c9d1d4303965-kube-api-access-sfs9b\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.757801 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.757692 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.757856 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.757818 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.859025 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.859000 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.859135 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.859045 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.859135 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.859087 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfs9b\" (UniqueName: \"kubernetes.io/projected/770f4175-e525-4a6f-b54d-c9d1d4303965-kube-api-access-sfs9b\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.859783 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.859758 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.861444 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.861424 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.884187 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.884128 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfs9b\" (UniqueName: \"kubernetes.io/projected/770f4175-e525-4a6f-b54d-c9d1d4303965-kube-api-access-sfs9b\") pod \"maas-api-revoke-sub-2e60ce57-w4sps\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:35.949310 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:35.949291 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:36.251837 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:36.251812 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps"] Nov 02 15:08:36.254669 ip-10-0-134-9 kubenswrapper[4431]: W1102 15:08:36.254641 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770f4175_e525_4a6f_b54d_c9d1d4303965.slice/crio-e4aa0c82d7cc1b709d330c66a4d53fe338a51c0bd18f3898958a8e74862279d9 WatchSource:0}: Error finding container e4aa0c82d7cc1b709d330c66a4d53fe338a51c0bd18f3898958a8e74862279d9: Status 404 returned error can't find the container with id e4aa0c82d7cc1b709d330c66a4d53fe338a51c0bd18f3898958a8e74862279d9 Nov 02 15:08:36.569108 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:36.569033 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" event={"ID":"770f4175-e525-4a6f-b54d-c9d1d4303965","Type":"ContainerStarted","Data":"e4aa0c82d7cc1b709d330c66a4d53fe338a51c0bd18f3898958a8e74862279d9"} Nov 02 15:08:37.575093 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:37.575056 4431 generic.go:356] "Generic (PLEG): container finished" podID="770f4175-e525-4a6f-b54d-c9d1d4303965" containerID="991735579b0c677ed9bd1e975b40e820d5f0b0d40e19292a74928232cb2949d5" exitCode=0 Nov 02 15:08:37.575477 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:37.575112 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" event={"ID":"770f4175-e525-4a6f-b54d-c9d1d4303965","Type":"ContainerDied","Data":"991735579b0c677ed9bd1e975b40e820d5f0b0d40e19292a74928232cb2949d5"} Nov 02 15:08:38.717243 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.717219 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:38.786011 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.785983 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770f4175-e525-4a6f-b54d-c9d1d4303965-kube-api-access-sfs9b\" (UniqueName: \"kubernetes.io/projected/770f4175-e525-4a6f-b54d-c9d1d4303965-kube-api-access-sfs9b\") pod \"770f4175-e525-4a6f-b54d-c9d1d4303965\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " Nov 02 15:08:38.786162 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.786144 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-xks-ca\") pod \"770f4175-e525-4a6f-b54d-c9d1d4303965\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " Nov 02 15:08:38.786240 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.786224 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-openshift-ca\") pod \"770f4175-e525-4a6f-b54d-c9d1d4303965\" (UID: \"770f4175-e525-4a6f-b54d-c9d1d4303965\") " Nov 02 15:08:38.786600 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.786567 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-openshift-ca" pod "770f4175-e525-4a6f-b54d-c9d1d4303965" (UID: "770f4175-e525-4a6f-b54d-c9d1d4303965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:08:38.788199 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.788171 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770f4175-e525-4a6f-b54d-c9d1d4303965-kube-api-access-sfs9b" pod "770f4175-e525-4a6f-b54d-c9d1d4303965" (UID: "770f4175-e525-4a6f-b54d-c9d1d4303965"). InnerVolumeSpecName "kube-api-access-sfs9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:08:38.788333 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.788259 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-xks-ca" pod "770f4175-e525-4a6f-b54d-c9d1d4303965" (UID: "770f4175-e525-4a6f-b54d-c9d1d4303965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:08:38.887176 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.887149 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:08:38.887176 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.887173 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfs9b\" (UniqueName: \"kubernetes.io/projected/770f4175-e525-4a6f-b54d-c9d1d4303965-kube-api-access-sfs9b\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:08:38.887329 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:38.887184 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770f4175-e525-4a6f-b54d-c9d1d4303965-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:08:39.586306 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:39.586276 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" Nov 02 15:08:39.586459 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:39.586309 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e60ce57-w4sps" event={"ID":"770f4175-e525-4a6f-b54d-c9d1d4303965","Type":"ContainerDied","Data":"e4aa0c82d7cc1b709d330c66a4d53fe338a51c0bd18f3898958a8e74862279d9"} Nov 02 15:08:39.586459 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:39.586361 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4aa0c82d7cc1b709d330c66a4d53fe338a51c0bd18f3898958a8e74862279d9" Nov 02 15:08:47.013432 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:47.013399 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:08:51.282152 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:08:51.282112 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:08:51.714335 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:51.714295 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:08:57.154630 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:08:57.154594 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:09:06.517485 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.517452 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn"] Nov 02 15:09:06.519848 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.518828 4431 cpu_manager.go:414] "containerMap: removing container" podUID="770f4175-e525-4a6f-b54d-c9d1d4303965" containerName="revoke-keys" Nov 02 15:09:06.519848 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.518851 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770f4175-e525-4a6f-b54d-c9d1d4303965" containerName="revoke-keys" Nov 02 15:09:06.519848 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.519054 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="770f4175-e525-4a6f-b54d-c9d1d4303965" containerName="revoke-keys" Nov 02 15:09:06.527304 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.527280 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.541586 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.541563 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn"] Nov 02 15:09:06.552647 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.552626 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:09:06.552761 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.552714 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 15:09:06.679382 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.679357 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.679519 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.679394 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.679519 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.679419 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2q4\" (UniqueName: \"kubernetes.io/projected/c6f709f5-b777-46a8-a4c2-539b4e9314cc-kube-api-access-zg2q4\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.780658 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.780594 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.780658 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.780629 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2q4\" (UniqueName: \"kubernetes.io/projected/c6f709f5-b777-46a8-a4c2-539b4e9314cc-kube-api-access-zg2q4\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.780826 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.780700 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.781225 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.781204 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.782877 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.782859 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.805737 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.805716 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2q4\" (UniqueName: \"kubernetes.io/projected/c6f709f5-b777-46a8-a4c2-539b4e9314cc-kube-api-access-zg2q4\") pod \"maas-api-revoke-sub-0af79180-4d7mn\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:06.840603 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:06.840586 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:07.135718 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:07.135697 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn"] Nov 02 15:09:07.138016 ip-10-0-134-9 kubenswrapper[4431]: W1102 15:09:07.137991 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f709f5_b777_46a8_a4c2_539b4e9314cc.slice/crio-51d376b87b353ec479eaefa70365490de92575c85cac159a8d159d9ea300c189 WatchSource:0}: Error finding container 51d376b87b353ec479eaefa70365490de92575c85cac159a8d159d9ea300c189: Status 404 returned error can't find the container with id 51d376b87b353ec479eaefa70365490de92575c85cac159a8d159d9ea300c189 Nov 02 15:09:07.768765 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:07.768669 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" event={"ID":"c6f709f5-b777-46a8-a4c2-539b4e9314cc","Type":"ContainerStarted","Data":"0baa05ac43bdba2a3624f39b6d144d3031b07d77e84f92095a3e02cdde757cda"} Nov 02 15:09:07.768765 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:07.768727 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" event={"ID":"c6f709f5-b777-46a8-a4c2-539b4e9314cc","Type":"ContainerStarted","Data":"51d376b87b353ec479eaefa70365490de92575c85cac159a8d159d9ea300c189"} Nov 02 15:09:07.814300 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:07.814250 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" podStartSLOduration=1.814235518 podStartE2EDuration="1.814235518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:07.812261219 +0000 UTC m=+4312.849366151" watchObservedRunningTime="2026-11-02 15:09:07.814235518 +0000 UTC m=+4312.851340449" Nov 02 15:09:08.774720 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:08.774687 4431 generic.go:356] "Generic (PLEG): container finished" podID="c6f709f5-b777-46a8-a4c2-539b4e9314cc" containerID="0baa05ac43bdba2a3624f39b6d144d3031b07d77e84f92095a3e02cdde757cda" exitCode=0 Nov 02 15:09:08.775141 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:08.774791 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" event={"ID":"c6f709f5-b777-46a8-a4c2-539b4e9314cc","Type":"ContainerDied","Data":"0baa05ac43bdba2a3624f39b6d144d3031b07d77e84f92095a3e02cdde757cda"} Nov 02 15:09:09.917658 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:09.917635 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:10.109313 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.109246 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-xks-ca\") pod \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " Nov 02 15:09:10.109313 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.109274 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f709f5-b777-46a8-a4c2-539b4e9314cc-kube-api-access-zg2q4\" (UniqueName: \"kubernetes.io/projected/c6f709f5-b777-46a8-a4c2-539b4e9314cc-kube-api-access-zg2q4\") pod \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " Nov 02 15:09:10.109494 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.109340 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-openshift-ca\") pod \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\" (UID: \"c6f709f5-b777-46a8-a4c2-539b4e9314cc\") " Nov 02 15:09:10.109748 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.109721 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-openshift-ca" pod "c6f709f5-b777-46a8-a4c2-539b4e9314cc" (UID: "c6f709f5-b777-46a8-a4c2-539b4e9314cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:09:10.111231 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.111208 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-xks-ca" pod "c6f709f5-b777-46a8-a4c2-539b4e9314cc" (UID: "c6f709f5-b777-46a8-a4c2-539b4e9314cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:09:10.111304 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.111280 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f709f5-b777-46a8-a4c2-539b4e9314cc-kube-api-access-zg2q4" pod "c6f709f5-b777-46a8-a4c2-539b4e9314cc" (UID: "c6f709f5-b777-46a8-a4c2-539b4e9314cc"). InnerVolumeSpecName "kube-api-access-zg2q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:09:10.210978 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.210952 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:09:10.211088 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.210978 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg2q4\" (UniqueName: \"kubernetes.io/projected/c6f709f5-b777-46a8-a4c2-539b4e9314cc-kube-api-access-zg2q4\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:09:10.211088 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.210995 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f709f5-b777-46a8-a4c2-539b4e9314cc-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:09:10.784715 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.784686 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" Nov 02 15:09:10.784913 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.784688 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af79180-4d7mn" event={"ID":"c6f709f5-b777-46a8-a4c2-539b4e9314cc","Type":"ContainerDied","Data":"51d376b87b353ec479eaefa70365490de92575c85cac159a8d159d9ea300c189"} Nov 02 15:09:10.784913 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:10.784827 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51d376b87b353ec479eaefa70365490de92575c85cac159a8d159d9ea300c189" Nov 02 15:09:16.883265 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.883231 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7"] Nov 02 15:09:16.883663 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.883646 4431 cpu_manager.go:414] "containerMap: removing container" podUID="c6f709f5-b777-46a8-a4c2-539b4e9314cc" containerName="revoke-keys" Nov 02 15:09:16.883663 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.883657 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f709f5-b777-46a8-a4c2-539b4e9314cc" containerName="revoke-keys" Nov 02 15:09:16.883746 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.883732 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f709f5-b777-46a8-a4c2-539b4e9314cc" containerName="revoke-keys" Nov 02 15:09:16.886931 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.886914 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:16.907225 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.907201 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7"] Nov 02 15:09:16.917574 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.917556 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:09:16.917688 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.917645 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 15:09:16.963763 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.963733 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnz98\" (UniqueName: \"kubernetes.io/projected/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-kube-api-access-hnz98\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:16.963913 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.963823 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:16.963980 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:16.963920 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:17.064719 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.064691 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:17.064878 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.064744 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnz98\" (UniqueName: \"kubernetes.io/projected/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-kube-api-access-hnz98\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:17.064878 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.064817 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:17.065591 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.065570 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:17.067010 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.066987 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:17.089460 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.089439 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnz98\" (UniqueName: \"kubernetes.io/projected/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-kube-api-access-hnz98\") pod \"maas-api-revoke-sub-023bd7c9-jh8n7\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:17.201720 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.201688 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:17.501144 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.501086 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7"] Nov 02 15:09:17.503365 ip-10-0-134-9 kubenswrapper[4431]: W1102 15:09:17.503336 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dc0b09_c469_4e09_8ef9_68c7f12814eb.slice/crio-f41d96448c9ff0a87134f883ee8645b5b698529b9803f2d7d2726a8ccf684c77 WatchSource:0}: Error finding container f41d96448c9ff0a87134f883ee8645b5b698529b9803f2d7d2726a8ccf684c77: Status 404 returned error can't find the container with id f41d96448c9ff0a87134f883ee8645b5b698529b9803f2d7d2726a8ccf684c77 Nov 02 15:09:17.819048 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.819002 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" event={"ID":"e9dc0b09-c469-4e09-8ef9-68c7f12814eb","Type":"ContainerStarted","Data":"172329c8ca427dfe1c51b5d4733e3b4dfb91300a7be64e6f6c1670e744ac79cf"} Nov 02 15:09:17.819180 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.819080 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" event={"ID":"e9dc0b09-c469-4e09-8ef9-68c7f12814eb","Type":"ContainerStarted","Data":"f41d96448c9ff0a87134f883ee8645b5b698529b9803f2d7d2726a8ccf684c77"} Nov 02 15:09:17.822288 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.822267 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:09:17.870896 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:17.870857 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" podStartSLOduration=1.870845563 podStartE2EDuration="1.870845563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:17.861568507 +0000 UTC m=+4322.898673438" watchObservedRunningTime="2026-11-02 15:09:17.870845563 +0000 UTC m=+4322.907950494" Nov 02 15:09:18.826840 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:18.826807 4431 generic.go:356] "Generic (PLEG): container finished" podID="e9dc0b09-c469-4e09-8ef9-68c7f12814eb" containerID="172329c8ca427dfe1c51b5d4733e3b4dfb91300a7be64e6f6c1670e744ac79cf" exitCode=0 Nov 02 15:09:18.827238 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:18.826867 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" event={"ID":"e9dc0b09-c469-4e09-8ef9-68c7f12814eb","Type":"ContainerDied","Data":"172329c8ca427dfe1c51b5d4733e3b4dfb91300a7be64e6f6c1670e744ac79cf"} Nov 02 15:09:19.966628 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:19.966599 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:19.990088 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:19.990065 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-openshift-ca\") pod \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " Nov 02 15:09:19.990184 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:19.990168 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-kube-api-access-hnz98\" (UniqueName: \"kubernetes.io/projected/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-kube-api-access-hnz98\") pod \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " Nov 02 15:09:19.990242 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:19.990227 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-xks-ca\") pod \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\" (UID: \"e9dc0b09-c469-4e09-8ef9-68c7f12814eb\") " Nov 02 15:09:19.990445 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:19.990425 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-openshift-ca" pod "e9dc0b09-c469-4e09-8ef9-68c7f12814eb" (UID: "e9dc0b09-c469-4e09-8ef9-68c7f12814eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:09:19.990556 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:19.990543 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:09:19.992192 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:19.992173 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-xks-ca" pod "e9dc0b09-c469-4e09-8ef9-68c7f12814eb" (UID: "e9dc0b09-c469-4e09-8ef9-68c7f12814eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:09:19.992250 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:19.992227 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-kube-api-access-hnz98" pod "e9dc0b09-c469-4e09-8ef9-68c7f12814eb" (UID: "e9dc0b09-c469-4e09-8ef9-68c7f12814eb"). InnerVolumeSpecName "kube-api-access-hnz98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:09:20.091030 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:20.090969 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnz98\" (UniqueName: \"kubernetes.io/projected/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-kube-api-access-hnz98\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:09:20.091030 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:20.090992 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dc0b09-c469-4e09-8ef9-68c7f12814eb-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:09:20.837144 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:20.837110 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" Nov 02 15:09:20.837331 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:20.837138 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023bd7c9-jh8n7" event={"ID":"e9dc0b09-c469-4e09-8ef9-68c7f12814eb","Type":"ContainerDied","Data":"f41d96448c9ff0a87134f883ee8645b5b698529b9803f2d7d2726a8ccf684c77"} Nov 02 15:09:20.837331 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:20.837195 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f41d96448c9ff0a87134f883ee8645b5b698529b9803f2d7d2726a8ccf684c77" Nov 02 15:09:21.282249 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:09:21.282215 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:09:28.205467 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:28.205433 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:09:32.848685 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:32.848646 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:09:49.713092 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:49.713047 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:09:51.345238 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:09:51.345198 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:09:56.830375 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.830344 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l"] Nov 02 15:09:56.830759 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.830725 4431 cpu_manager.go:414] "containerMap: removing container" podUID="e9dc0b09-c469-4e09-8ef9-68c7f12814eb" containerName="revoke-keys" Nov 02 15:09:56.830759 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.830736 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9dc0b09-c469-4e09-8ef9-68c7f12814eb" containerName="revoke-keys" Nov 02 15:09:56.830862 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.830829 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9dc0b09-c469-4e09-8ef9-68c7f12814eb" containerName="revoke-keys" Nov 02 15:09:56.833190 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.833174 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:56.853723 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.853698 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l"] Nov 02 15:09:56.864551 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.864527 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:09:56.864681 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.864629 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 15:09:56.961997 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.961973 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:56.962125 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.962003 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwww\" (UniqueName: \"kubernetes.io/projected/f2c5472c-fa98-4fd8-a909-ddf46048801e-kube-api-access-tqwww\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:56.962125 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:56.962036 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:57.063263 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:57.063231 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwww\" (UniqueName: \"kubernetes.io/projected/f2c5472c-fa98-4fd8-a909-ddf46048801e-kube-api-access-tqwww\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:57.063393 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:57.063312 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:57.063453 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:57.063439 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:57.064098 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:57.064076 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:57.065635 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:57.065616 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:57.088283 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:57.088228 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwww\" (UniqueName: \"kubernetes.io/projected/f2c5472c-fa98-4fd8-a909-ddf46048801e-kube-api-access-tqwww\") pod \"maas-api-revoke-sub-b6df3cf4-fdn5l\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:57.145098 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:57.145075 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:09:57.439692 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:57.439664 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l"] Nov 02 15:09:57.442476 ip-10-0-134-9 kubenswrapper[4431]: W1102 15:09:57.442447 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c5472c_fa98_4fd8_a909_ddf46048801e.slice/crio-f1f4fde558128c4f4feeedf1deb1c010d2044b0ed971c3ad0c06f90f6e53962f WatchSource:0}: Error finding container f1f4fde558128c4f4feeedf1deb1c010d2044b0ed971c3ad0c06f90f6e53962f: Status 404 returned error can't find the container with id f1f4fde558128c4f4feeedf1deb1c010d2044b0ed971c3ad0c06f90f6e53962f Nov 02 15:09:58.089822 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:58.089733 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" event={"ID":"f2c5472c-fa98-4fd8-a909-ddf46048801e","Type":"ContainerStarted","Data":"fc8fd286cc1bef7402d951b9a5e93525f98cab1a74c1ef403c67da533e2789b8"} Nov 02 15:09:58.089822 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:58.089798 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" event={"ID":"f2c5472c-fa98-4fd8-a909-ddf46048801e","Type":"ContainerStarted","Data":"f1f4fde558128c4f4feeedf1deb1c010d2044b0ed971c3ad0c06f90f6e53962f"} Nov 02 15:09:58.133597 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:58.133551 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" podStartSLOduration=2.133536564 podStartE2EDuration="2.133536564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:58.131882126 +0000 UTC m=+4363.168987063" watchObservedRunningTime="2026-11-02 15:09:58.133536564 +0000 UTC m=+4363.170641496" Nov 02 15:09:59.095605 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:59.095572 4431 generic.go:356] "Generic (PLEG): container finished" podID="f2c5472c-fa98-4fd8-a909-ddf46048801e" containerID="fc8fd286cc1bef7402d951b9a5e93525f98cab1a74c1ef403c67da533e2789b8" exitCode=0 Nov 02 15:09:59.096068 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:09:59.095660 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" event={"ID":"f2c5472c-fa98-4fd8-a909-ddf46048801e","Type":"ContainerDied","Data":"fc8fd286cc1bef7402d951b9a5e93525f98cab1a74c1ef403c67da533e2789b8"} Nov 02 15:10:00.231865 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.231843 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:10:00.289485 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.289463 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-openshift-ca\") pod \"f2c5472c-fa98-4fd8-a909-ddf46048801e\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " Nov 02 15:10:00.289623 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.289492 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c5472c-fa98-4fd8-a909-ddf46048801e-kube-api-access-tqwww\" (UniqueName: \"kubernetes.io/projected/f2c5472c-fa98-4fd8-a909-ddf46048801e-kube-api-access-tqwww\") pod \"f2c5472c-fa98-4fd8-a909-ddf46048801e\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " Nov 02 15:10:00.289623 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.289559 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-xks-ca\") pod \"f2c5472c-fa98-4fd8-a909-ddf46048801e\" (UID: \"f2c5472c-fa98-4fd8-a909-ddf46048801e\") " Nov 02 15:10:00.289890 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.289864 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-openshift-ca" pod "f2c5472c-fa98-4fd8-a909-ddf46048801e" (UID: "f2c5472c-fa98-4fd8-a909-ddf46048801e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:10:00.291580 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.291549 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-xks-ca" pod "f2c5472c-fa98-4fd8-a909-ddf46048801e" (UID: "f2c5472c-fa98-4fd8-a909-ddf46048801e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:10:00.291689 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.291634 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c5472c-fa98-4fd8-a909-ddf46048801e-kube-api-access-tqwww" pod "f2c5472c-fa98-4fd8-a909-ddf46048801e" (UID: "f2c5472c-fa98-4fd8-a909-ddf46048801e"). InnerVolumeSpecName "kube-api-access-tqwww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:00.390928 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.390904 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:10:00.391031 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.390928 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c5472c-fa98-4fd8-a909-ddf46048801e-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:10:00.391031 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:00.390942 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqwww\" (UniqueName: \"kubernetes.io/projected/f2c5472c-fa98-4fd8-a909-ddf46048801e-kube-api-access-tqwww\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:10:01.105358 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:01.105326 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" event={"ID":"f2c5472c-fa98-4fd8-a909-ddf46048801e","Type":"ContainerDied","Data":"f1f4fde558128c4f4feeedf1deb1c010d2044b0ed971c3ad0c06f90f6e53962f"} Nov 02 15:10:01.105358 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:01.105346 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6df3cf4-fdn5l" Nov 02 15:10:01.105547 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:01.105370 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1f4fde558128c4f4feeedf1deb1c010d2044b0ed971c3ad0c06f90f6e53962f" Nov 02 15:10:07.208566 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.208536 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75"] Nov 02 15:10:07.208997 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.208958 4431 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5472c-fa98-4fd8-a909-ddf46048801e" containerName="revoke-keys" Nov 02 15:10:07.208997 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.208971 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5472c-fa98-4fd8-a909-ddf46048801e" containerName="revoke-keys" Nov 02 15:10:07.209092 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.209045 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5472c-fa98-4fd8-a909-ddf46048801e" containerName="revoke-keys" Nov 02 15:10:07.212293 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.212271 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.234068 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.234041 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75"] Nov 02 15:10:07.241215 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.241191 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:10:07.241336 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.241228 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nkklw\"" Nov 02 15:10:07.360094 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.360067 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.360216 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.360103 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.360265 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.360241 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54dp\" (UniqueName: \"kubernetes.io/projected/fae2d67e-8b10-4f41-8006-93fdfb245862-kube-api-access-s54dp\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.461317 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.461257 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54dp\" (UniqueName: \"kubernetes.io/projected/fae2d67e-8b10-4f41-8006-93fdfb245862-kube-api-access-s54dp\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.461317 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.461301 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.461454 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.461325 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.461939 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.461920 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.463590 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.463573 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.486200 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.486172 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54dp\" (UniqueName: \"kubernetes.io/projected/fae2d67e-8b10-4f41-8006-93fdfb245862-kube-api-access-s54dp\") pod \"maas-api-revoke-sub-c8177d20-r4w75\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.525405 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.525382 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:07.821465 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:07.821414 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75"] Nov 02 15:10:07.823771 ip-10-0-134-9 kubenswrapper[4431]: W1102 15:10:07.823739 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae2d67e_8b10_4f41_8006_93fdfb245862.slice/crio-89aa7a2b30c6cbccac5ba0229070ab328d0f21772cf7ffe3fc33aea3d24d3a00 WatchSource:0}: Error finding container 89aa7a2b30c6cbccac5ba0229070ab328d0f21772cf7ffe3fc33aea3d24d3a00: Status 404 returned error can't find the container with id 89aa7a2b30c6cbccac5ba0229070ab328d0f21772cf7ffe3fc33aea3d24d3a00 Nov 02 15:10:08.114282 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:08.114255 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:10:08.137858 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:08.137825 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" event={"ID":"fae2d67e-8b10-4f41-8006-93fdfb245862","Type":"ContainerStarted","Data":"cd721100cc829d71a3362750b74a33d2132dba18d30c4372543622a7ce22161f"} Nov 02 15:10:08.137972 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:08.137885 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" event={"ID":"fae2d67e-8b10-4f41-8006-93fdfb245862","Type":"ContainerStarted","Data":"89aa7a2b30c6cbccac5ba0229070ab328d0f21772cf7ffe3fc33aea3d24d3a00"} Nov 02 15:10:08.181262 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:08.181221 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" podStartSLOduration=1.181209059 podStartE2EDuration="1.181209059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:08.180070506 +0000 UTC m=+4373.217175439" watchObservedRunningTime="2026-11-02 15:10:08.181209059 +0000 UTC m=+4373.218313991" Nov 02 15:10:09.145625 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:09.145589 4431 generic.go:356] "Generic (PLEG): container finished" podID="fae2d67e-8b10-4f41-8006-93fdfb245862" containerID="cd721100cc829d71a3362750b74a33d2132dba18d30c4372543622a7ce22161f" exitCode=0 Nov 02 15:10:09.146166 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:09.145657 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" event={"ID":"fae2d67e-8b10-4f41-8006-93fdfb245862","Type":"ContainerDied","Data":"cd721100cc829d71a3362750b74a33d2132dba18d30c4372543622a7ce22161f"} Nov 02 15:10:10.475958 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.475931 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:10.545107 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.545075 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-xks-ca\") pod \"fae2d67e-8b10-4f41-8006-93fdfb245862\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " Nov 02 15:10:10.545392 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.545371 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae2d67e-8b10-4f41-8006-93fdfb245862-kube-api-access-s54dp\" (UniqueName: \"kubernetes.io/projected/fae2d67e-8b10-4f41-8006-93fdfb245862-kube-api-access-s54dp\") pod \"fae2d67e-8b10-4f41-8006-93fdfb245862\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " Nov 02 15:10:10.545596 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.545580 4431 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-openshift-ca\") pod \"fae2d67e-8b10-4f41-8006-93fdfb245862\" (UID: \"fae2d67e-8b10-4f41-8006-93fdfb245862\") " Nov 02 15:10:10.546417 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.546392 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-openshift-ca" pod "fae2d67e-8b10-4f41-8006-93fdfb245862" (UID: "fae2d67e-8b10-4f41-8006-93fdfb245862"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:10:10.549511 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.549470 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-xks-ca" pod "fae2d67e-8b10-4f41-8006-93fdfb245862" (UID: "fae2d67e-8b10-4f41-8006-93fdfb245862"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:10:10.549933 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.549909 4431 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae2d67e-8b10-4f41-8006-93fdfb245862-kube-api-access-s54dp" pod "fae2d67e-8b10-4f41-8006-93fdfb245862" (UID: "fae2d67e-8b10-4f41-8006-93fdfb245862"). InnerVolumeSpecName "kube-api-access-s54dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:10:10.646578 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.646493 4431 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s54dp\" (UniqueName: \"kubernetes.io/projected/fae2d67e-8b10-4f41-8006-93fdfb245862-kube-api-access-s54dp\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:10:10.646578 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.646526 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-openshift-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:10:10.646578 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:10.646546 4431 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae2d67e-8b10-4f41-8006-93fdfb245862-maas-api-xks-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Nov 02 15:10:11.157159 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:11.157129 4431 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" Nov 02 15:10:11.157340 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:11.157122 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8177d20-r4w75" event={"ID":"fae2d67e-8b10-4f41-8006-93fdfb245862","Type":"ContainerDied","Data":"89aa7a2b30c6cbccac5ba0229070ab328d0f21772cf7ffe3fc33aea3d24d3a00"} Nov 02 15:10:11.157340 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:11.157263 4431 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89aa7a2b30c6cbccac5ba0229070ab328d0f21772cf7ffe3fc33aea3d24d3a00" Nov 02 15:10:18.591151 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:18.591116 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ghltt"] Nov 02 15:10:21.282212 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:10:21.282172 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:10:51.281846 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:10:51.281809 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:10:59.987882 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:10:59.987844 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7xw9r_49b694a0-4451-451b-b77b-f2c1774f478f/manager/0.log" Nov 02 15:11:00.446998 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:00.446971 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fttqq_05fb164b-a69a-4241-97d0-3705e3233692/manager/0.log" Nov 02 15:11:01.004038 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:01.004010 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4kw4d_e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550/server/0.log" Nov 02 15:11:01.212810 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:01.212766 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2nb2_bb8c30b8-0631-48c2-8e5e-f7feae79214c/manager/0.log" Nov 02 15:11:02.879482 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:02.879450 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-023bd7c9-jh8n7_e9dc0b09-c469-4e09-8ef9-68c7f12814eb/revoke-keys/0.log" Nov 02 15:11:03.073831 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:03.073802 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0af79180-4d7mn_c6f709f5-b777-46a8-a4c2-539b4e9314cc/revoke-keys/0.log" Nov 02 15:11:03.261037 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:03.261007 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-277d08f7-s6wks_496ef5f8-7a43-4f6b-8e0a-366aa83be69c/revoke-keys/0.log" Nov 02 15:11:03.464180 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:03.464153 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2e60ce57-w4sps_770f4175-e525-4a6f-b54d-c9d1d4303965/revoke-keys/0.log" Nov 02 15:11:03.863641 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:03.863613 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-78cad57d-c7ktw_f5a3d55c-a10c-487f-8d73-0b6c8a370967/revoke-keys/0.log" Nov 02 15:11:04.066402 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:04.066372 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b6df3cf4-fdn5l_f2c5472c-fa98-4fd8-a909-ddf46048801e/revoke-keys/0.log" Nov 02 15:11:04.281263 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:04.281233 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c8177d20-r4w75_fae2d67e-8b10-4f41-8006-93fdfb245862/revoke-keys/0.log" Nov 02 15:11:14.616753 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:14.616723 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-6x7qk_5665c19f-2391-4df6-b139-a7021a3285c8/postgres/0.log" Nov 02 15:11:21.282187 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:11:21.282149 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:11:27.185412 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:27.185380 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mgjcj_1b17862c-afb6-4a79-a21f-54b8e5ea3063/kuadrant-console-plugin/0.log" Nov 02 15:11:27.704967 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:27.704942 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ghltt_4dd71c7a-a298-456d-8d7e-0b1884d987ed/limitador/0.log" Nov 02 15:11:29.562670 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:29.562640 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8f9c48f5f-prw2d_33771e58-bf16-4c4d-bb9c-ac49aa5d26df/kube-auth-proxy/0.log" Nov 02 15:11:29.992464 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:29.992435 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-2m48q_f771e0c6-2f81-4e2f-afd3-20be3bc4eddf/istio-proxy/0.log" Nov 02 15:11:31.044725 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:31.044695 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-q8r79_bc27444d-3855-4031-8b67-8fe90fb4ae96/istio-proxy/0.log" Nov 02 15:11:31.266788 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:31.266756 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-t9gjp_fefae9cc-8fdb-4c2a-9029-5218b3f2ea97/istio-proxy/0.log" Nov 02 15:11:51.281377 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:11:51.281341 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:11:51.537429 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:51.537324 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ndjmx_dc38ff14-bc2b-49aa-b37e-a32248814b35/global-pull-secret-syncer/0.log" Nov 02 15:11:52.332817 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:52.332764 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5q6hs_e3a1d83b-52ea-4fcd-9843-4eb8199d3d65/konnectivity-agent/0.log" Nov 02 15:11:52.768269 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:11:52.768239 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-9.ec2.internal_6e4f9ec4cb1fc6480c545594116d855c/haproxy/0.log" Nov 02 15:12:00.038210 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:00.038181 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mgjcj_1b17862c-afb6-4a79-a21f-54b8e5ea3063/kuadrant-console-plugin/0.log" Nov 02 15:12:00.380656 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:00.380574 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ghltt_4dd71c7a-a298-456d-8d7e-0b1884d987ed/limitador/0.log" Nov 02 15:12:01.695839 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.695806 4431 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8"] Nov 02 15:12:01.696438 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.696371 4431 cpu_manager.go:414] "containerMap: removing container" podUID="fae2d67e-8b10-4f41-8006-93fdfb245862" containerName="revoke-keys" Nov 02 15:12:01.696438 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.696388 4431 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae2d67e-8b10-4f41-8006-93fdfb245862" containerName="revoke-keys" Nov 02 15:12:01.696612 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.696498 4431 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae2d67e-8b10-4f41-8006-93fdfb245862" containerName="revoke-keys" Nov 02 15:12:01.699633 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.699616 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.707062 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.707041 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-lib-modules\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.707140 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.707077 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-proc\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.707140 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.707110 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lgx\" (UniqueName: \"kubernetes.io/projected/a1e830b8-3888-406a-bacc-5cbcf2e95303-kube-api-access-28lgx\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.707140 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.707134 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-podres\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.707254 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.707164 4431 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-sys\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.718368 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.718348 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcxf8\"/\"openshift-service-ca.crt\"" Nov 02 15:12:01.720484 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.720460 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8"] Nov 02 15:12:01.731701 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.731676 4431 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcxf8\"/\"kube-root-ca.crt\"" Nov 02 15:12:01.767589 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.767569 4431 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fcxf8\"/\"default-dockercfg-pmlxg\"" Nov 02 15:12:01.807772 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.807748 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lgx\" (UniqueName: \"kubernetes.io/projected/a1e830b8-3888-406a-bacc-5cbcf2e95303-kube-api-access-28lgx\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.807908 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.807816 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-podres\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.807908 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.807875 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-sys\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.808025 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.807922 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-lib-modules\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.808025 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.807963 4431 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-proc\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.808108 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.808024 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-sys\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.808108 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.807960 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-podres\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.808108 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.808026 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-proc\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.808108 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.808075 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e830b8-3888-406a-bacc-5cbcf2e95303-lib-modules\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.832337 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.832312 4431 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lgx\" (UniqueName: \"kubernetes.io/projected/a1e830b8-3888-406a-bacc-5cbcf2e95303-kube-api-access-28lgx\") pod \"perf-node-gather-daemonset-m9fr8\" (UID: \"a1e830b8-3888-406a-bacc-5cbcf2e95303\") " pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:01.926554 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:01.926530 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22ae8630-0de4-45c5-86e7-ae97eef1e60b/alertmanager/0.log" Nov 02 15:12:02.012093 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.012024 4431 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:02.056456 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.056428 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22ae8630-0de4-45c5-86e7-ae97eef1e60b/config-reloader/0.log" Nov 02 15:12:02.184725 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.184690 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22ae8630-0de4-45c5-86e7-ae97eef1e60b/kube-rbac-proxy-web/0.log" Nov 02 15:12:02.307353 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.307328 4431 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8"] Nov 02 15:12:02.309683 ip-10-0-134-9 kubenswrapper[4431]: W1102 15:12:02.309656 4431 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda1e830b8_3888_406a_bacc_5cbcf2e95303.slice/crio-97e76a1129c1a99805244c4e615ad769ef8f90726ed05ed01b040cb4eb82f057 WatchSource:0}: Error finding container 97e76a1129c1a99805244c4e615ad769ef8f90726ed05ed01b040cb4eb82f057: Status 404 returned error can't find the container with id 97e76a1129c1a99805244c4e615ad769ef8f90726ed05ed01b040cb4eb82f057 Nov 02 15:12:02.313026 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.313005 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22ae8630-0de4-45c5-86e7-ae97eef1e60b/kube-rbac-proxy/0.log" Nov 02 15:12:02.441527 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.441496 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22ae8630-0de4-45c5-86e7-ae97eef1e60b/kube-rbac-proxy-metric/0.log" Nov 02 15:12:02.568703 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.568634 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22ae8630-0de4-45c5-86e7-ae97eef1e60b/prom-label-proxy/0.log" Nov 02 15:12:02.696666 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.696630 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22ae8630-0de4-45c5-86e7-ae97eef1e60b/init-config-reloader/0.log" Nov 02 15:12:02.754320 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.754287 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" event={"ID":"a1e830b8-3888-406a-bacc-5cbcf2e95303","Type":"ContainerStarted","Data":"cb23cbaa5ca9d100e99dde4239003fa4923d305a1b0bb55f6260a089bbb01051"} Nov 02 15:12:02.754493 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.754341 4431 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" event={"ID":"a1e830b8-3888-406a-bacc-5cbcf2e95303","Type":"ContainerStarted","Data":"97e76a1129c1a99805244c4e615ad769ef8f90726ed05ed01b040cb4eb82f057"} Nov 02 15:12:02.754493 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.754366 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:02.799180 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.799127 4431 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" podStartSLOduration=1.799113324 podStartE2EDuration="1.799113324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 15:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:02.79776772 +0000 UTC m=+4487.834872654" watchObservedRunningTime="2026-11-02 15:12:02.799113324 +0000 UTC m=+4487.836218255" Nov 02 15:12:02.972449 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:02.972424 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rwvx2_8088b390-9d13-4cfa-903a-68ac3f2a81a4/kube-state-metrics/0.log" Nov 02 15:12:03.096510 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:03.096488 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rwvx2_8088b390-9d13-4cfa-903a-68ac3f2a81a4/kube-rbac-proxy-main/0.log" Nov 02 15:12:03.247701 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:03.247632 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rwvx2_8088b390-9d13-4cfa-903a-68ac3f2a81a4/kube-rbac-proxy-self/0.log" Nov 02 15:12:03.640966 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:03.640937 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9jmvm_361c7df3-61c9-4cdb-8c3b-4d59fba75b1d/node-exporter/0.log" Nov 02 15:12:03.766603 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:03.766578 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9jmvm_361c7df3-61c9-4cdb-8c3b-4d59fba75b1d/kube-rbac-proxy/0.log" Nov 02 15:12:03.891230 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:03.891174 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9jmvm_361c7df3-61c9-4cdb-8c3b-4d59fba75b1d/init-textfile/0.log" Nov 02 15:12:04.785866 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:04.785823 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s99vd_197100a3-c701-4a29-a7de-de6df973e373/kube-rbac-proxy-main/0.log" Nov 02 15:12:04.912899 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:04.912869 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s99vd_197100a3-c701-4a29-a7de-de6df973e373/kube-rbac-proxy-self/0.log" Nov 02 15:12:05.042470 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:05.042396 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-s99vd_197100a3-c701-4a29-a7de-de6df973e373/openshift-state-metrics/0.log" Nov 02 15:12:05.181117 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:05.181090 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac83895e-0242-4235-970e-eb73c99fa77f/prometheus/0.log" Nov 02 15:12:05.307699 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:05.307638 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac83895e-0242-4235-970e-eb73c99fa77f/config-reloader/0.log" Nov 02 15:12:05.439261 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:05.439233 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac83895e-0242-4235-970e-eb73c99fa77f/thanos-sidecar/0.log" Nov 02 15:12:05.565544 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:05.565483 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac83895e-0242-4235-970e-eb73c99fa77f/kube-rbac-proxy-web/0.log" Nov 02 15:12:05.692717 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:05.692691 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac83895e-0242-4235-970e-eb73c99fa77f/kube-rbac-proxy/0.log" Nov 02 15:12:05.818499 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:05.818426 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac83895e-0242-4235-970e-eb73c99fa77f/kube-rbac-proxy-thanos/0.log" Nov 02 15:12:05.944848 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:05.944825 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac83895e-0242-4235-970e-eb73c99fa77f/init-config-reloader/0.log" Nov 02 15:12:06.083266 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:06.083200 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-nv82m_9d022dc5-98a1-4834-8fd9-749f05411719/prometheus-operator/0.log" Nov 02 15:12:06.205506 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:06.205479 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-nv82m_9d022dc5-98a1-4834-8fd9-749f05411719/kube-rbac-proxy/0.log" Nov 02 15:12:06.334060 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:06.334007 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-mngff_f26a21ee-590a-451c-b9fa-5d4f87d61c2a/prometheus-operator-admission-webhook/0.log" Nov 02 15:12:08.424078 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:08.424054 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7xw9r_49b694a0-4451-451b-b77b-f2c1774f478f/manager/0.log" Nov 02 15:12:08.771120 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:08.771044 4431 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fcxf8/perf-node-gather-daemonset-m9fr8" Nov 02 15:12:09.003607 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:09.003580 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fttqq_05fb164b-a69a-4241-97d0-3705e3233692/manager/0.log" Nov 02 15:12:09.140654 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:09.140628 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4kw4d_e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550/server/0.log" Nov 02 15:12:10.111677 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:10.111646 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2nb2_bb8c30b8-0631-48c2-8e5e-f7feae79214c/manager/0.log" Nov 02 15:12:11.187358 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:11.187333 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-687865856c-95sjw_605e26d1-e6bb-4f8e-8b33-3c9277139663/console/0.log" Nov 02 15:12:11.332415 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:11.332386 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-t9b9x_2f686f4c-ccfa-4a39-a9c3-a26794765b2b/download-server/0.log" Nov 02 15:12:11.876136 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:11.876107 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7xw9r_49b694a0-4451-451b-b77b-f2c1774f478f/manager/0.log" Nov 02 15:12:12.423045 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:12.423010 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fttqq_05fb164b-a69a-4241-97d0-3705e3233692/manager/0.log" Nov 02 15:12:12.602755 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:12.602725 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4kw4d_e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550/server/0.log" Nov 02 15:12:13.602106 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:13.602075 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2nb2_bb8c30b8-0631-48c2-8e5e-f7feae79214c/manager/0.log" Nov 02 15:12:15.149598 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:15.149568 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7xw9r_49b694a0-4451-451b-b77b-f2c1774f478f/manager/0.log" Nov 02 15:12:15.227876 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:15.227821 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bv8r5_ef68447e-3ad3-4dd2-b353-4aa00584d583/dns/0.log" Nov 02 15:12:15.353213 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:15.353188 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bv8r5_ef68447e-3ad3-4dd2-b353-4aa00584d583/kube-rbac-proxy/0.log" Nov 02 15:12:15.680215 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:15.680177 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fttqq_05fb164b-a69a-4241-97d0-3705e3233692/manager/0.log" Nov 02 15:12:15.746626 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:15.746600 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kjcwb_c568d04d-8bb4-41bc-ad89-ba2f09a1c3e5/dns-node-resolver/0.log" Nov 02 15:12:15.829673 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:15.829647 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4kw4d_e4d7afae-cb3c-4ae4-b4ae-f6f725ecd550/server/0.log" Nov 02 15:12:16.865856 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:16.865802 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z2nb2_bb8c30b8-0631-48c2-8e5e-f7feae79214c/manager/0.log" Nov 02 15:12:17.492409 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:17.492378 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dtc82_a3206ad3-d2a7-4eec-81cb-db3a09dee305/node-ca/0.log" Nov 02 15:12:19.473023 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:19.472994 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8f9c48f5f-prw2d_33771e58-bf16-4c4d-bb9c-ac49aa5d26df/kube-auth-proxy/0.log" Nov 02 15:12:19.615244 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:19.615211 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-2m48q_f771e0c6-2f81-4e2f-afd3-20be3bc4eddf/istio-proxy/0.log" Nov 02 15:12:19.972440 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:19.972408 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-q8r79_bc27444d-3855-4031-8b67-8fe90fb4ae96/istio-proxy/0.log" Nov 02 15:12:20.041047 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:20.041017 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-t9gjp_fefae9cc-8fdb-4c2a-9029-5218b3f2ea97/istio-proxy/0.log" Nov 02 15:12:21.236336 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:21.236309 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8f9c48f5f-prw2d_33771e58-bf16-4c4d-bb9c-ac49aa5d26df/kube-auth-proxy/0.log" Nov 02 15:12:21.282642 ip-10-0-134-9 kubenswrapper[4431]: E1102 15:12:21.282608 4431 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vwqt2" containerName="extractor" Nov 02 15:12:21.513174 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:21.513104 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-2m48q_f771e0c6-2f81-4e2f-afd3-20be3bc4eddf/istio-proxy/0.log" Nov 02 15:12:22.206934 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:22.206910 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-q8r79_bc27444d-3855-4031-8b67-8fe90fb4ae96/istio-proxy/0.log" Nov 02 15:12:22.338115 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:22.338090 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55fbd65575-t9gjp_fefae9cc-8fdb-4c2a-9029-5218b3f2ea97/istio-proxy/0.log" Nov 02 15:12:23.502684 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:23.502657 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mgjcj_1b17862c-afb6-4a79-a21f-54b8e5ea3063/kuadrant-console-plugin/0.log" Nov 02 15:12:23.681099 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:23.681070 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ghltt_4dd71c7a-a298-456d-8d7e-0b1884d987ed/limitador/0.log" Nov 02 15:12:25.430711 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:25.430677 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4q2dh_b9d5b255-ab45-4fce-9226-a02f7ffec197/serve-healthcheck-canary/0.log" Nov 02 15:12:27.292195 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:27.292157 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vwqt2_166c406a-eb7f-41ce-9d49-71f3fda95c35/kube-rbac-proxy/0.log" Nov 02 15:12:27.419550 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:27.419524 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vwqt2_166c406a-eb7f-41ce-9d49-71f3fda95c35/exporter/0.log" Nov 02 15:12:27.618566 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:27.618500 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vwqt2_166c406a-eb7f-41ce-9d49-71f3fda95c35/extractor/0.log" Nov 02 15:12:31.500945 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:31.500917 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-d84mp_cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290/manager/0.log" Nov 02 15:12:31.627094 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:31.627065 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-d84mp_cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290/copy-manifests/0.log" Nov 02 15:12:33.959550 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:33.959520 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-597b95cb84-74djh_3f153a84-0da6-47a5-aaea-26fc51b053ba/cert-manager-controller/0.log" Nov 02 15:12:34.102094 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:34.102069 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-mj5lp_0d666d72-faf5-4172-bcb5-04db28685143/cert-manager-webhook/0.log" Nov 02 15:12:34.192279 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:34.192253 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-fqd5g_e2532e4b-b7e8-4a9c-b300-7f008758df4a/jobset-operator/0.log" Nov 02 15:12:35.135264 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:35.135229 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-d84mp_cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290/manager/0.log" Nov 02 15:12:35.202027 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:35.202002 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-78cdb466dc-d84mp_cd1ff8c9-abc2-4f9c-8f4f-d8f60177e290/copy-manifests/0.log" Nov 02 15:12:35.618883 ip-10-0-134-9 kubenswrapper[4431]: I1102 15:12:35.618850 4431 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7xw9r_49b694a0-4451-451b-b77b-f2c1774f478f/manager/0.log"